Engineering a better way to hire programmers

New data suggest that, when it comes to evaluating software development skills, seeing is better than believing

A main writing code on a whiteboard
Credit: Q Family CC BY 2.0

The length of the hiring process for tech jobs (in fact, for most jobs in the U.S.) has increased significantly over the last five years. Much of the increase is due to additional screening methods being used by employers, particularly for more complex jobs, such as software engineers. Companies will often rule prospective developer hires in (or out) based on a variety of things like education, prior experience, and the ability to solve coding problems on a whiteboard. One new tech recruiting company, however, feels there are better ways to hire developers and they’ve got data suggesting they may be on to something.

Triplebyte is a recruiting company which began operations earlier this year that specializes in identifying software developers with all-around programming know-how. Believing that current methods for hiring programmers don’t work well, the company has developed a protocol for evaluating the general programming abilities of developers which is centered around using a clear scoring system, being as transparent with candidates as possible, and making them feel comfortable during the process. In its first month of operation, the company fully vetted 300 candidates in a 30-day period and, recently, published an analysis of the process and results. To find out more, I also spoke via email with Harj Taggar, Triplebyte’s CEO and co-founder, about the process.

Currently, Triplebyte’s evaluation process involves four steps, starting with an online technical screen consisting of a FizzBuzz-like coding test and a multiple-choice programming quiz. After that comes a 15-minute phone call to discuss a past technical project, and then a 45-minute screen-sharing interview where the candidate writes code. Finally, the last step is a longer 2-hour screen-sharing coding session to work on a larger project. For the coding portions, candidates program on their own equipment, use the language and dev tools of their choice, and select from a small handful of coding projects to work on.

While they’ve just begun to try their theories in the real world, they’ve already seen some interesting results. Specifically…

Success in coding tests didn’t correlate with coding skill

Triplebyte looked at the correlation between the correctness and quality of a candidate’s solution to a FizzBuzz-type test and his or her success in the 45-minute coding interview.  According to Taggar, candidates are given a score for the latter that “looks at a number of factors e.g. progress made on the problem, cleanliness of code, depth of understanding of concepts behind the problem.” The correlation between solving the FizzBuzz-like test correctly and scoring well in the coding interview was less than 20%.

Performance on a programming quiz was a strong predictor of interview success

The second portion of the initial online screen, a multiple-choice quiz which involved looking at code, was much more positively correlated with success in the 45-minute interview. In particular, a composite score, based on the number of questions answered correctly and the time-to-finish (faster being better) had a 54% correlation for the subsequent interview score.

Sounding knowledgable about past technical projects was a poor indicator of coding ability

Triplebyte found that a candidate’s ability to speak intelligently about past technical projects, even complex ones, was a weak predictor of coding skill. Specifically, the company only found a 20% correlation between a candidate being rated as impressive based on discussing past experiences and doing well in the 45-minute coding session.

There are a lot of caveats with these results, of course. For one, it’s a non-scientific study with a small sample size. Secondly, as Taggar told me, “We don't test for specific knowledge of particular languages, just general programming ability.” Evaluation of more specific programming-skills are left to companies to which Triplebyte refers qualified programmer candidates.

Taggar told me it was “too early for us to have meaningful data” on the number of candidates that Triplebyte passed who were subsequently hired and performed successfully. However, the company plans to track success-on-the-job metrics in the future to help better refine their evaluation process. Taggar told me that their client companies will also continue to have input into the process. “A number of them have provided valuable feedback and we're constantly comparing our process with the companies we send candidates too. We believe heavily in listening to feedback and iterating our process,” he wrote.

Feedback from candidates has been positive, according to Taggar, who told me, “Candidates love that (1) they can use their preferred language and development environment, (2) choose from a selection of problems (3) have a cordial/friendly relationship with their interviewer.”

Interestingly, it seems to me that the Triplebyte process may actually help reduce the increasingly lengthy time-to-hire developer candidates face. Taggar said they’re generally able to complete the evaluation of a candidate in 48 hours (depending, of course, on his or her availability). Also, companies considering these candidates can skip general programming skill screening, and focus right away on skills specific to their shop, which could help to speed up the interview process.

Whatever the outcome of Triplebyte’s approach it’s great to see a company taking a rigorous approach to the evaluation of software developers. Besides, who knows? Maybe the days of nervous candidates being asked to work out problems on a whiteboard are numbered.

ITWorld DealPost: The best in tech deals and discounts.
Shop Tech Products at Amazon