|All I need is the right coding test and|
The right candidate will pop out.
In my view a coding test should be used to answer at least the following questions
- Is the candidate's understanding of English (say) good enough to let them do the job?
- Does their coding style fit the company needs and/or culture of their prospective team
- Are they a “get it done” person or a “get it right” person.
- If they are unable to complete the test would they be able to perform in the workplace if mentored
- If they do not complete the test does this indicate an inability to handle stress? If so does it matter?
And this cannot be achieved with remote timed tests administered by a third party.
One simple test is to ask them to write fizzbuzz, whereby for integers between (say) 1 and 100 inclusive the applicant should produce code that will print “Fizz” if the number is a multiple of three, “Buzz” if the number is a multiple of five and “FizzBuzz” if the number is a multiple of both.
As an exercise I programmed fizzbuzz using a Java 8 stream in order to further internalise the Java 8 functional paradigm. I started with a slightly different algorithm to the obvious one that first came to mind and, in the spirit of a timed test, reverted to the simpler algorithm at the first sign of difficulty. This is another risk of a timed test: the candidate will run with the first solution they think of and this may be neither optimal or creative. There is a universal tendency to think of a superior solution two minutes after the test ends.
Another problem with tests is that a candidate who performs well may be a details person to the extent that they cannot see the wood for the trees and, once in post, will miss design problems in your codebase
David Saintloth  notes that FizzBuzz type tests test knowledge of the minutia of a given programming language and that syntax errors in coding are often punished harshly even if ti is clear the candidate understands the algorithm they wish to use. Further he says
fizz/buzz tests ONLY test programming (and in that not very well either). What people want to hire though are engineers not programmers....but they test for programmers, job requests are programmer focused (listing a flurry of buzzwords) which leaves potentially brilliant engineers out in the cold.
Ericsson  notes that aptitude tests, and coding tests are a form of aptitude test, predict short term performance but not long term or even medium term performance. Coding tests pander to the tendency for developers to focus on tiny details rather then the big picture. As an example I noted in a post on code reviews  that a comment in  on how code reviews focussed on a minor and irrelevant aspect of the code was followed by a large number of comments discussing the code presented totally hijacking the discussion: this is a bit like judging a painter by a single brush stroke rather than the picture as a whole and how to hold the brush rather than what do to with it.
Some people say that rather than coding tests candidates should be asked to show code samples or discuss projects they have undertaken. These solutions have problems. Code produced for an employer or client is often covered by a non disclosure agreement and some developers deliberately refrain from spare time coding for a number of reasons: for example to avoid burnout or domestic friction, to study other things, architecture for example or something totally unrelated to technology. The argument that such programmers are not “Passionate” about technology is specious, involves the “Real Programmer” syndrome and that the developer is, or should be or become autistic as a result of obsession with technology. This of course benefits managers trying to squeeze more out of their resources before throwing them away when they burn out .
If coding tests are to be used the employer must decide what they want to learn from these tests rather then just applying them as a silver bullet to fix the probably unsolvable problem of picking if not the best candidate, then one capable of doing the job. It is possible that in the future AI techniques will be better able than humans to pick the “right” candidate but such an AI could and probably would, pick another AI to do the job.
More seriously filtering candidates on the basis of automated or semi automated screening not only risks leaving out potentially brilliant candidates but ignores the heuristics that cultural fit is more important than technical ability, that technical weaknesses can be trained out and that a hire who turns out unable to do the job for which they hired may do brilliantly in another role and that may only be found out after they are hired.
There is a role for coding tests but using them to screen applicants automatically suggests that the fabled shortage of developers is indeed a fable: If there were a real shortage employers would be willing to take on and train candidates they now reject.
- Ericsson,Krampe and Tesch-Romer: The Role of Deliberate Practice in the Acquisition of Expert Performance: PsychologicalReview 1993, Vol. 100. No. 3, 363-406
- Code reviews: do we really need them: http://digitalsteampunk.blogspot.com/2016/03/code-reviews-do-we-need-them.html
- https://phabricator.wikimedia.org/T114419 Ways to make code review not suck
- Karojisatsu: Mental health and suicide risks posed by IT culture for Managers and Developers http://digitalsteampunk.blogspot.com/2016/01/karojisatsu-mental-health-and-suicide.html