What is the concept behind this “10x Engineer?”
Wouldn’t it be great if we could find the perfect person for every position every time?
Yes, it would
We all want that, but are we geared towards doing that?
I am not questioning intentions here, merely questioning the means
Before I ponder further, I would like us to be on the same page as to who falls under the 10X Engineer category
10X Engineer a.k.a the coding Ninja a.k.a rockstar developer is an engineer who contributes 10X times the value of one engineer.
Don’t get me wrong. They are not developers who can type out the code at 10X speed but those who can build solutions that are 10X effective.
10X Engineers build and love to build better software.
I love the way Joel Spolsky, CEO, and co-founder of StackOverflow puts this into perspective:
|
|
Now that we have established the idea of a 10X Engineer, keep in mind that a 10X Engineer that suits my team might not be the 10X engineer that suits yours.
Why do we fail in hiring 10X Engineers?
We fail because we are assessing things the wrong way. We rely too much on resumes, textbook-oriented assessments, unstructured interviews.
The good news is that you and I are not alone; even the best of companies including Google have chinks in their recruitment process. Here is a famous example of Google rejecting Max Howell, creator of HomeBrew, as he could not invert a binary tree on a whiteboard
So, what is the right way?Fortunately for us, Frank L Schmidt and John E Hunter chose 19 assessment methods and did a detailed study on how accurate each of the assessment methods is in predicting candidates’ on-the-job performance
I accept the research does have some limitations and the data was not only from the tech fraternity. But is anyway better than relying on opinions, anecdotes, and hunches
This is a table of the top 12 assessment methods reverse sorted based on their validity scores on how well each test predicts a candidate’s on-the-job performance
| Assessment type | Validity Score |
| Work sample assessments | 0.54 |
| General mental ability tests | 0.51 |
| Structured Interviews | 0.51 |
| Peer Ratings | 0.49 |
| Job knowledge tests | 0.48 |
| Integrity tests | 0.41 |
| Unstructured Interviews | 0.38 |
| Assessment centers | 0.37 |
| Biographical data measures | 0.35 |
| Conscientiousness tests | 0.31 |
| Reference checks | 0.26 |
| Job experience (years) | 0.18 |
| Years of education | 0.10 |
| Interests | 0.10 |
The right way!
Of course, using all these tests will help us get better clarity on identifying our unicorn – the 10X Engineer. But owing to the limited time, money, and other resources, we can’t use them all. We need to choose
Our all-time favorites: unstructured interviews and resume highlights (Job experience in years, education, interests, etc.) are seen in the bottom half
And clearly, Work sample assessments have the best individual performance, followed by Structured Interviews, and General Mental Ability(GMA) tests.
We are overconfident about our expertise and experience in judging people that we dislike, realigning ourselves to more structured approaches fearing that these assessments might outsource human judgment to a machine
But one thing we seem to miss is these structured evaluations and automation are there to supplement human judgment with data and not replace them.
One big issue
GMA tests tend to attract legal issues, at least in the US, as it is biased against minorities. Here is one such example – Griggs vs. Duke Power Co.
By using GMA, your company might not only have to pay a hefty penalty but would also run the risk of being biased.
Our quest for hiring 10X Engineers might get severely derailed if we are perceived as a biased organization
So we now have the top two:
- Work sample assessments and
- Structured Interviews
Getting started
Before getting started with the assessments, conduct a detailed job analysis that will help us implement these assessments.
1. Work sample assessments
Work sample assessments are designed to mirror a situation parallel to that at work, under realistic and standardized conditions. For example, deploying an API, debugging code, or building a machine learning model
Here is a what Iris Bohnet, the author of What Works: Gender Equality by Design, has to say about work sample assessments
“The best indicator of future job performance is to give the candidate a work sample test; a task that the candidate will be doing on the job” |
|
These are ideally designed as projects and are implemented as take-home projects that the candidates work on
Even with such overwhelming validity, work sample assessments are being used only by a handful of companies because
- Traditionally, work sample assessments incur high cost to implement as it involves one engineering source for every candidate code evaluation.
