Because the vast majority of job interviews are with terrible candidates, even if the majority of candidates are excellent. This apparent paradox has a simple explanation: excellent candidates selectively apply to a few companies and get interviews/offers at almost all of them. On the other hand, terrible candidates are rejected at every step of the hiring process, and have to constantly reenter the interview pool.
Suppose 90% of candidates are excellent and 10% are terrible. If the excellent 90% only need to interview at one company, whereas the bad 10% need to interview at 20 companies, then only 0.9/(0.1*20+0.9)=31% of interviews will be with qualified candidates. To retierate: almost 70% of interviews will be with terrible candidates, even though 90% of people applying for jobs are excellent.
Because the cost of a bad hire is so consequential, the interview process is not designed to efficiently handle a minority of qualified candidates, but rather efficiently weed out a majority of horrible candidates. It is therefore a terrible process for the people actually qualified to pass it.
Why put the blame only on candidates? Interviewers are equally bad to interviewees. I have been to both sides of the table and can guarantee that 80% of interviewers would not be fit for my job or the process of hiring.
I don't think the parent comment meant to 'blame' the candidates; I read this as a statistical picture. Because of how the numbers work out, the market (as measured per-interview) is flooded with bad candidates. This does not disagree with the fact that companies are _also_ usually pretty bad at interviewing.
Same story, I think. Well-paid positions at sensible low drama companies are filled quickly, while companies with glaring issues may interview and make offers to dozens of candidates before finding one who accepts the offer. So as a candidate you also see a disproportionate number of bad interviews.
Your point is interesting but I don’t see how it answers the question.
“The majority of interviews being statistically rejections “is not obviously related to a need to make the interview process more demanding than the actual job - Testing for the role would also weed out bad candidates.
Here’s some alternatives:
- engineers testing for the idealized version of their job, rather than the realities of it - “a true engineer must know how to balance a binary tree! Nevermind that I spend my time dealing with support tickets regarding a null pointer that slipped by in a code review”.
- companies using long processes as negotiation tactics - “you went through two months of trials so you’re less likely to reject our offer now and start over”
- design by committee - “every interested party and team must give an approval in their own step”
- interview as marketing - “see? We deal with tough challenges, this is an interesting place to work at”.
As someone interviewing for the first time in a long time I can tell you most assuredly that there are a disproportionate number of awful interviewers too.
Of the interviews I've had I would say about 3/4 have tried to catch you out with some inane gotcha that you would never see in the wild or have a very specific solution in mind without exploring or discussing. Sometimes both.
I can attest. I've personally been shut out of two positions because I didn't say the "magic word" that the interviewer was looking for. In one instance, I kept using laymen's terms to describe an optical engineering process, because I knew they weren't optical engineers (far from it). They wanted me to say the precise technical phrase, but wouldn't explicitly ask that; they just kept asking questions around that phrase.
In another job where I was hired, I was tortured for over 10 minutes as one interviewer tried to get me to say the magic word. Fortunately, his manager was present, and shut down the questioning at that point. I never did learn what word the subordinate was trying to get me to say, but I was fully qualified for the job, and hired.
My more recent interviews have been fine, but for awhile I had several that were (in the words of xkcd) "communicating badly then acting smug when you're misunderstood".
We have to acknowledge that most companies are horrible at recognizing a bad hire and then appropriately handling the situation. A lot of companies profess to "hire fast, fire fast" but very few actually do. Until that changes, the cost of a "bad hire" will continue to be disproportionately high.
> Because the cost of a bad hire is so consequential,
This is stated all the time and I feel it is true. But is there any way to make it less consequential? That was my main argument for contract-to-hire (though I know there are downsides to that approach).
Are there any other ways to make hiring less risky?
> Are there any other ways to make hiring less risky?
Professional licensing.
Many other fields require professional licenses. I don't understand there's so much opposition in our field.
(Ok, I do understand.) In general, licensing has some risks:
The lemons will get excluded from the field. (Which is kind of the point.)
Or, the lemons will decide what the criteria for a professional license is; which turns it into a BS hurdle.
---
That being said, the article gets closer to the point of what a professional license is for: "An interview is like running 100m and a job is like a 10k.". If the license is more like running a 10k, then interviewers can rely on it to do a better screening than they could ever hope to do.
The issue with professional licensing, is that it's very, very specific.
In things like civil engineering, there's usually mandated context. You have to work within certain parameters, so it's not too difficult to test with real-world criteria.
With software engineering, it's all over the place. In fact, one of the most exciting things that I used to look for, in potential candidates, was people who were not bogged down with dogma, and would bring alternative viewpoints to the team.
Since anyone coming into my team would require a ton of training; regardless of their seniority, I always had a nice, long on-ramp, in which I could evaluate people.
There can be a halfway solution that helps: A professional software engineering license that signifies an extremely basic, barebones level of skill. Companies would still need to interview candidates, but they wouldn't have to do FizzBuzz or "write a for loop" types of interviews with candidates who literally cannot write code at all. It wouldn't guarantee the person was an expert in inverting binary trees, but it would at least represent a minimal knowledge and skill bar that one would expect any software engineer to meet.
One of my interview questions used to involve putting up a parameterized SQL statement on the whiteboard, (select userid, email from users where userid=?), give the candidate instructions on how to use a data reader, and then expect them to construct a "user" object.
I did this because I was part of a project that failed because the leads did not know how to work with a database. They thought the ORM would just magically load objects. Without understanding the basic limitations of data readers, they painted themselves into a corner.
Licensing basically needs to filter this kind of thing. IE, you don't get a license to work with a SQL database unless you can work with a data reader.
---
The irony was that we very rarely touched SQL, but the question was one of the greatest filters out there! It showed who understood basic concepts, who was just BSing their way though, and who would get stuck in high-level abstractions.
One additional downside to professional licensing is that it can be time consuming and costly. For example, to bring a new physician on board to a practice they must be:
* Licensed (allowed to practice in that state)
* Credentialed (degrees and experience verified)
* Enrolled (able to be imbursed via insurance)
* Priviledged (authorized to perform certain tasks/roles)
This could take weeks or months per physician and there is usually an entire team (Medical Staff Office/MSO) dedicated to the work.
> This could take weeks or months per physician and there is usually an entire team
That seems like a better alternative to what we have now! Some companies search forever for the "perfect" candidate, or otherwise have to sift through so much SPAM resumes that they are already taking "weeks or months."
In my experience, the hiring managers with the best track record have these in common, which mostly boil down to the hiring teams doing their homework:
(1) The expectations for the position are clearly defined
(2) The hiring team members coordinate on questions and expected responses, and they are consistent in interviews.
(3) The hiring team members know how to spot potential issues (e.g., excessive bad-mouthing of previous employers, etc.)
(4) The hiring teams effectively leverage their networks for references. Ideally, there are not-too-distant trusted relationships between the candidate and the hiring team. In the absence of this, references are followed up on carefully (this has become an art form in modern times.)
These reduce the risk of someone slipping through the cracks. Hiring teams also get better with experience, so any mistakes should be carefully analyzed and corrective actions incorporated into the hiring process.
After interviews, you hire Alice but Bob was a close second. Bob goes to a competitor. Alice doesn’t work out. A double whammy. Why not immediately try to get Bob? I’ve only seen this a few time.
In the US at least it seems common to review performance and put certain benefits on hold for 90 days to see if it's working out. This wouldn't mitigate the costs of the initial hiring or the opportunity cost of not selecting another candidate, though.
The 'cost of a bad hire' is received wisdom that needs to go away. The first order effects of your team's time investment are easy to see and make good content for your engineering leadership blog when you're aiming for promotion. The second order effects are what get debated in threads like this ad infinitum.
Paradoxically, a higher bar for hiring increases these consequences for everyone. A bad hire is only consequential in the first place because hiring managers are slow to cut them loose. Managers are slow to cut loose because they are morally culpable for the consequences to the individual they hired. When a manager extends an offer, they are accepting some responsibility for a significant change in a person's life. It's very difficult to walk that back when it's a bad fit, knowing that hiring is a slow process and every other company out there is scared of making a bad choice. But at the end of the day, interviews are an approximation of the candidate/company fit in what is ultimately a matching problem. More attempts make for better matches. Companies and candidates both would be better served by being faster to hire and faster cut loose.
This is closely related to the claim "We only hire the top 1%". Yes, but when your competitors reject a candidate they don't just stop existing, eventually they choose you.
Yes... Joel Spolsky puts it a bit uncharitably here [1]:
> I’m exaggerating a lot, but the point is, when you select 1 out of 200 applicants, the other 199 don’t give up and go into plumbing (although I wish they would… plumbers are impossible to find). They apply again somewhere else, and contribute to some other employer’s self-delusions about how selective they are.
I mean this is certainly an effect, but largely outdated reasoning. Every tech company you can think of copied verbatim a Google-style interview process out of complete reflex without understanding largely why. Now you discover that your candidate, who can invert a tree just fine, and can regurgitate quicksort, is totally useless at delivering software, managing stakeholders, or understanding the business logic.
If I remember correctly, there's an ancient article by Joel Spolsky about this.
That great candidates are not out there doing blind interviews, only those of us who are average are interviewing and going through hoops. Engineers below average are almost constantly interviewing.
It’s rather older than this. Any argument about “the market is swamped by bad X in circulation and good X are much rarer in the market than you would naively expect” (software developers, online dating, used cars, debased currency…) is just a version of Gresham’s Law, which is 500 years old: https://en.m.wikipedia.org/wiki/Gresham%27s_law
> Because the vast majority of job interviews are with terrible candidates, even if the majority of candidates are excellent. This apparent paradox has a simple explanation: excellent candidates selectively apply to a few companies and get interviews/offers at almost all of them. On the other hand, terrible candidates are rejected at every step of the hiring process, and have to constantly reenter the interview pool.
Do you have actual data showing this? Or is this just your intuition?
Because if it's the latter, my intuition is pretty different.
No one is an "excellent candidate" for every position, and many people who are "excellent candidates" for a given position might not even recognize that excellence in themselves. Therefore, they're not going to be only applying for positions at the very best companies; they're going to be applying for any position they think they have a chance at, assuming they think they can actually be OK with the job (eg, they might not want to apply for a job in adtech if they are personally repulsed by the ethics of surveillance capitalism).
Additionally, your scenario sounds like it paints the candidate pool for jobs in general as a bimodal distribution, with one peak of "terrible candidates" and one peak of "excellent candidates", with very little in the middle. My intuition says that it's much more likely to resemble a normal distribution.
No; what's much, much more likely is that most people are decent candidates for many jobs in their field, and excellent candidates for a few, but their chances of actually getting the opportunity to apply to those few (between the position being open and them searching at the same time, and them finding out that it's open) are slim, so they have to apply for a great many of the positions that they're only decent candidates for. That means that they'll try many times before finding something. This can lead to a lot of frustration and even desperation, creating a willingness to engage in some shady techniques to actually get a human to talk to you and recognize your value.
Then there are a few people who are, indeed, nothing but shady techniques, and they are likely to flood all channels with as many AI-generated or otherwise low-quality applications as they can manage.
So no; even if most applications any position sees are terrible applications, most interviews are likely to be with decent-but-not-excellent candidates, and most people are still going to have to interview with a few or even tens of companies before they are actually offered a position.
In my experience, technical interviews are not really useful past the very basic "Can this candidate actually write a conditional and has the slightest clue about programming?". Ability to solve hard leetcode-like problems under time pressure in a stressful environment doesn't meaningfully translate to "will be a great contributor to the team work on the kind of problem we have".
Our best hires are nearly always coming from the network of a team member or people we contracted with and decided to hire full time.
Most of my time in interview nowadays is spent understanding what the candidate has done before, explaining to them what we do and asking open questions to see how they would approach our issues and how they link them to their experience. If it seems to fit, we hire. My country standard contract offers a fairly long probation period for new hire and we don't hesite about parting with people when it's not working after a quarter. We are very explicit about this policy.
> Do you have any great candidates who approach you and then, finding out your approach, pass?
We never generalised the contractors to employee thing. It's just that we hire contractors fairly often to fill in temporary needs and we generally extend offer for full time employment to the ones we would like to keep with us when we can. They generally say no because they like being contractors.
For candidate applying, I have yet to see one explicitely refuse an offer because they know we don't always keep going after the probation period. Then again, the standard probation period for engineers in my country is 3 months which can be renewed once so they would get the same offer anywhere.
But I have never seen someone being surprised when we parted after the 3 months. If someone told me they were during our offboarding meeting, I would take it as us having done something seriously wrong during our onboarding.
To be honest, we don't have that many hirings which end up not working and of them I can only remember one being due to an actual lack of skill from someone who clearly padded their resume yet managed to go through the interview. Some people needed a bit more time than others to reach the level of delivery we expect but that's ok. We don't always need rock stars, just professional who can deliver consistently and are open to learning new things. When we need specialists and don't have someone sufficently skilled internally, we contract. Working together with a specialist tends to raise the level of the team as a whole.
Most what I consider our true hiring failures have come from a mismatch between what the person expected and what the job actually is. That's why I now take time to ensure the person I'm interviewing actually has a good idea of what we are doing.
I find problem-solving questions always pretty good at weeding out candidates. Give them a real world problem with a fictional situation from their future job and just ask how they would tackle the problem.
Invert a binary tree, find the optimal path in an n-dimensional space of real numbers given by a function, design a horizontally scalable build system, get grilled by at least two people on your leadership skills... then, if you're extremely lucky, get hired to configure the button to be blue and have round corners. And get laid off in 9 months.
(Not saying this happened to me but it's a common story I've heard in the last few years)
I agree with the author that it is hard to assess someone's skill if you have a list of 100 people to interview and you know nothing about them. The bigger the stack of applications the easier it is to treat them like data and not people.
I know plenty of software developers who write, maintain, or contribute to OSS libraries; they write blog posts, give talks at conferences/meetups, and make videos in their spare time (or sometimes as part of their work). I've rarely walked into an interview where the hiring manager or the technical interviewer hadn't just read my name off my resume as I joined the meeting. Get to know the candidate's work before assuming they know nothing!
Maybe people involved in the hiring process should be given more time to properly research a candidate. Relying on ATS' and putting the burden on applicant's to do the work of proving themselves is causing a lot of folks to burnout just trying to get a job.
Jm2c but interviews tell you absolutely none, nothing, about what kind of a professional the candidate is.
I have no clue whether he'll care and help or pretend to work and drag everybody else down.
There's a huge number of incredibly capable developers who could pass any interview but then spend days playing video games and sabotaging projects and teams.
I really don't believe in technical interviews, I'd rather base the relationship on trust, if you tell me you're good/experienced at X I trust you to be. If it was bs you'll be shown the door with ease.
Instead many companies make it insanely hard to get you hired, but also incredibly hard to cut you out even if you're impact is a very net negative.
This works very well for contractors, less so for full-time employees. You can't just fire somebody on a whim, at least not for free, not even in the US, let alone in most of Europe.
To be clear, I'm not saying worker protections are bad, just that if firing is much more expensive than hiring, you can't really afford to hire any warm body that walks through the door. These days everyone and their mother is in CS, there are many more talented people than ever, but also more duds than ever.
Yes, you can even in Europe where you have high protections like in Italy, there's a probation period. Standard is 3 months, higher seniority can be even 6.
It's also the reason why e.g. Italians don't change jobs for 10/20% higher salary, you don't want to go through probation.
IME: the only valuable signal comes from direct personal referals. If you have someone who you think is good, and they recommend someone who they say is good from a previous engagement, odds are it will work out. There's a transitive, holistic measure in play; they're not going to destroy their reputation by recommending a weak player or even a strong jerk. The problem here is scale (you quickly milk direct networks dry) but nothing else seems to work well.
It's the reason why most jobs, even in Europe, have a probation period. During this oeriod firing is inexpensive. In my current employment the probation period was 6 months.
If in 6 months you still can't figure out if a hire was good or not, interviewing won't save you.
> Jm2c but interviews tell you absolutely none, nothing, about what kind of a professional the candidate is.
With the caveat that I have not been on the hiring side of things myself, my feeling about this is that it can—if a) the person you're interviewing is acting fully in good faith, b) they're not the type of person who gets so nervous about interviews that it skews their whole personality temporarily, and c) your interview process is actually decently-designed (I don't think leetcode or any type of "gotcha"-questions will give you a good picture of how professional a candidate is, nor will an interview for any kind of technical position designed entirely by nontechnical people with no domain input).
The big, big problem is the genuinely small proportion of people who come into a job interview in bad faith, seeking not to demonstrate but to hide their true skill and personality, because (rightly or wrongly) they believe it will hinder their chances of a good position.
The alternative I prefer isn't any of the given ones, but a scaled interview. I start with something that you can solve with a Python one-liner, if you know what you're doing, but if you solve that instantly I've got a series of questions that will scale up until we're eventually doing the task using untrusted user input, outputting into a security-sensitive environment, under heavy load that we want to be cheap, in a highly-available environment, etc. etc. I also tell the applicant up front that this is the plan, because if you don't do that, it feels like you're jerking them around by constantly changing the requirements. I've also got a number of directions it can go depending on the applicant, and depending on what they tell me about their past experience.
The other nice thing about this is your interviewee generally doesn't leave feeling like a failure; it's not like I have three questions and you can get them all wrong. Unfortunately there are some people who end up spinning on the very easiest question for the entire session, and, uh, well... I can only do so much, if you really don't know anything about programming at all. This is at least the exception, though.
I have not had to do an interview in the age of practical AI yet, though. In person I don't think I'd have to change much, I've always interviewed with a policy of "I'm not worried about whether the string split command takes its parameters in this or that order, I just want to know you know it exists" and I can basically serve as an AI in the same way I was already being the API reference. Remotely, I'm not sure what I'd do yet.
I do like your approach more than many alternatives but there are still lots of challenges. Two big issues: 1. this is not typically how even a confident, prepared, solid developer approaches coding problems, so they can be poorly prepared for this style of interview. The result is you don't get an accurate representation of the individual. 2. it takes an awful lot of the interviewer's time to prepare, even if you can scale to multiple candidates with much of the same work. You need a very strong technical resource to deliver this style of interview, and they could probably be doing higher value work.
I've hired a lot of the past ~5 years and my take away: it sucks for everyone - on both sides - except the early HR-led functions, and they don't add any value or much help. They may actually make most of the challenges much worse.
"this is not typically how even a confident, prepared, solid developer approaches coding problems, so they can be poorly prepared for this style of interview. The result is you don't get an accurate representation of the individual."
It may not be how you approach coding problems but it tends to accurately reflect how people program in real life, so I don't think it's all that much of a stretch to ask people to show it in an interview. Nobody gets an assignment to write a scalable, testable, blah blah blah 100 pages of requirements project and just sits down and writes it linearly from start to finish. Everything is incrementally developed.
"it takes an awful lot of the interviewer's time to prepare,"
Perhaps in general, but not for me past the first couple of times. However, that's because I do strongly agree with...
"You need a very strong technical resource to deliver this style of interview"
Yes, definitely.
However I don't believe there's a way around this. I don't believe you can get high-quality results for technical hiring from any possible "interview" you can hand to a non-technical programmer.
Moreover I don't even know why people think this is or should be possible. Other than the popular "I want it therefore it should be possible" argument, which the universe doesn't particularly respect. I wouldn't expect that even armed with a prepared formal interview that I could interview an industrial chemist, oil rig engineer, or a plumber. There still seems to be a certain amount of residual contempt for programming in some circles and that includes the idea that a non-programmer should be expected to be able to interview a programmer at anything more than a super-basic level. No, programming is at this point as technical as any other career out there. It really wasn't 30 years ago, but it is now, where the basic standard for the simplest possible project now encompasses a suite of technologies and competencies just to get started and the complexity goes up from there.
Also, while a senior engineer perhaps shouldn't be doing full time interviewing, interviewing is extremely high value work. It determines the course of the entire team for years at a time. The interviews I did years ago still determine the course of my day-to-day activity. Huge impact.
That is a really neat interview format, the lightning round of varying themes of common tasks! This right here proves to me you are a good interviewer:
> I'm not worried about whether the string split command takes its parameters in this or that order, I just want to know you know it exists
I’ve run quite a few “can you write code” interviews in the age of practical AI, and I don’t know if I’ve been lucky, am good at breaking through nonsense, or if internet claims are exaggerated, but I can hardly tell the difference between now and the before times. You get someone on a call, you explain a problem, you see how they approach it, you probe along the way. I don’t work for a giant FAANG-like, maybe that’s part of it.
Valid points, but I think the most obvious reason is that (at least the recognizable) employers are swamped with applications that all look great on paper, which likely got much worse with the rise of good LLMs. Compared with similar high paying carreers, like medicine (multi-year residency) or high finance (solving math and probability problems on the spot), the hiring process for software engineers isn't especially gruesome.
> A doctor who has a diploma is much more likely to be a useful/good doctor than a person with a "computer science" diploma will be a good developer.
A few things to consider:
What we practice is "software engineering." Computer science is closely related to software engineering; but not the same thing. (It's like the difference between a degree in physics vs mechanical engineering.)
Doctors still have to be board certified, which requires self-study of topics that aren't taught in class. Some people do get their medical diploma and fail their board certifications, in which case they can't practice.
I should add: One of things, for doctors, that board certification does is make sure that doctors know topics that aren't covered in school.
My computer science degree overlooked a lot of topics that are very important when developing an industrial strength application; specifically, an application that is not an academic exercise, which gets back to:
> A doctor who has a diploma is much more likely to be a useful/good doctor than a person with a "computer science" diploma will be a good developer.
If we, as an industry, had something similar to doctors getting board certified, we could put pressure on ourselves (and academic institutions) to make sure that ample opportunity is provided in schools to learn how to be a good developer.
Back when I started, I think maybe half of the people I worked with had a relevant degree at most. Even that might be an overestimate.
Back then, CS degree programs didn't even teach higher level languages than C/++ as they were thought to change too quickly. Whatever you learned during your four years wouldn't apply after you graduated. Instead, the programs focused on the low level implementation details with the theory that was where the engineering and science were.
Now, the same school I went to has courses and tracks for web technologies, so who knows.
Companies keep ignoring the historical interview point. I have been to a few occasions when companies needed the exact thing i built in the past (e.g. migration to clickhouse), but chose to put me into a random take home related to a different technology (e.g. some bigquery assignment) and eventually reject me. Go off script and ask me details about the project which might solve your hands, why do i need to talk about something else?
quick answer: this takes time & energy; the interviewing company is willing to miss you to save this effort. Plus the interviewers are not likely to be good at these tasks, even if they're solid developers; it's a skill they have relatively little practice and training.
I understand this for a dry candidate pool. I have been there as a hiring manager, you need some signal with take home or live interviewing. But on the rare opportunity that you find someone who is willing to talk in details about a similar project you want to do, you skip the pipeline and if he indeed built it, you hire him.
Interviews allow just a little time for strangers to attempt to sync up and get a sense of each other.
Actual jobs offer much more time for people to learn to sync and communicate.
Also, on both sides of the interview table, people have varying strengths when it comes to short and long term communication skills. Plenty of interviewers are not good at interviewing, just as plenty of candidates are not good at their side of the process.
In short, interviews are a very poor approach to choosing who belongs long term.
Ultimately, regardless of the interview process used, every job is ultimately a long term “try and see” interview. You only know how someone fits by trying them for a while.
I tended to do the "Historical Interview" one. In my case, it worked well.
I think take-home tests are probably damn near worthless, these days. People will just feed the assignment into an LLM, and return the results.
I am not a fan of LeetCode, because I don't want to waste time, studying stuff that won't have any relevance to what needs to get done. I like to keep my dance card full, and wasting precious time to make someone else happy, isn't my idea of a good time.
That said, I understand why they are there. I just feel that they aren't really something that I'd use to judge senior-level talent; which was what I used to hire.
> Asking a candidate to solve some problem which lets a candidate work in a slightly less high stakes environment. but requires candidates to do extra work, taking longer than just the interview.
I refuse a lot of these. I've put a lot of time into many of them, just to be ghosted.
Typically, when the take-home comes with a laundry list of frameworks and 3rd party libraries, I walk away. Coming up to speed with all of them is too time consuming for the probability of getting ghosted.
It takes a lot of work on the interviewers side to come up with a good take-home. It requires discipline to reject the temptation to just throw in a laundry list of requirements.
I know it is hearsay, but it seems many times interviews try to get free work from candidates. Here, have a look at this real problem, see if you can solve it in whatever the interview time frame is.
When there are too many candidates and not so many jobs, and absolutely no regulation, employers are free to exploit applicants without consequences. Employers will do this even when there is no actual job being offered, so they can use the availability of applicants to apply downward pressure on the salaries of the workers they already have.
I have never seen this from either side. Think about it from the hiring side: how good of a solution could be produced by someone with zero context and experience in the business, domain and internal environment? This would be like less efficient "organic" vibe coding.
Interviews have been extensively studied by phd, figuring out what works and what doesn't. I have yet to see anyone write a blog or comment that shows even an awareness that this research exists, much less what it in it. (I know this research exists, but I'll admit to not knowing how to find it. My company insists the interview process they make me follow is based on it)
Cargo culting, and the fear of bad hires. Everyone wants "the best engineers". Probably 1% of tech firms work with deep tech, where boundaries are pushed, and every percentage of improvement or degradation can result in non-trivial gains or losses. The rest can do just fine with normal engineers, and do interviews like the rest of the professional world.
The 100m / 10k analogy is fantastic, but the reality is even worse.
Let’s say you expect a candidate to stay for average five years. That’s about 1200 working days. If you get to interview them for 1 day, then that day is 8.3m to the 10k. You get to watch them run for 8.3m and decide how well they can run the 10k.
You're not even watching them for the skills required on the job either. Most jobs aren't even about running at all.
Basic, boring, line-of-business web application shops are asking candidates to implement Boyer-Moore and k-d trees. When the applicant is going to be hired to mostly update JSX templates and figure out why the customer is frustrated. It's like every business needs Olympic athletes when they're running a hot-dog stand.
It's one thing if your business is writing a RTOS for a custom platform you develop. Asking a candidate to implement a slab allocator might be a fine exercise. It fits the work being done.
Businesses need to get off their high horse and look in the mirror. Most of them are doing mediocre things that need average, good-natured people who are willing to work.
I think there's also another element that the author is missing and that is likability. I've never experienced this, but I've heard about people being put through grueling interview processes merely to weed them out because they were deemed "not a good culture fit".
I think at least half of all interviews are a popularity contest, and the other half are your qualifications.
Not a fan on hiring based on popularity, but I do wish companies spent more time on actual culture fit and whether or not the person they are hiring is a raging asshole. I've worked with my share of "brilliant jerks" and they're just awful to be around. Even if they are John Carmack level of coding, it's not worth it--they are a net negative because of things like poor attitude, lack of communication skill, arrogance, my-way-or-the-highway stance on everything... I wish that filtering these people out was 50% of my company's interviews!
And I can tell from experience that it's quite easy to get rid of "brilliant jerks" during the recruitement process.
The key is to make them stop the process, not you; so you won't have to deal with HR or management telling you that they are brilliant and you should not reject them just because "you" don't like their behavior.
Scaring jerks out is quite easy: just be excessively nice during the interview. Tell them how much time and energy everybody take caring for each other's well-being and feelings.
Tell them that in the team you have lunch together every single day.
And you do stuff together after works.
And during the weekends.
They'll just run.
Sure, you're lying because you're working in a regular team where people are simply respectful professionnals, but they don't need to know that.
You'll do everybody's a favour. The jerk included.
I don't think "popularity contest" is accurate, but cultural fit is both a real - and important - component AND a risky opportunity to apply massive personal bias. Lots of variability here; I have a coworker who asked the exact questions to all candidates to avoid this, while I can't stop going off script. I know my risk is higher but I also find better candidates (while very likely passing on very good candidates for the wrong reasons).
Is there any other industry with such a disconnect between education and job where the candidates need to do another exam when they try to join the job?
A major issue the author missed is it is much harder to fire a non-performer in the tech industry today.
It takes 2 quarters (ie. 6 months) to go from recognizing a problem employee to firing said employee.
This makes the risk of hiring the wrong candidate significant as a hiring manager, because a bad hire reflects badly on you and eats up your budget thus preventing a backfill.
On top of that, firing individual employees can lead to litigation risk (even if frivolous), thus requiring hiring managers to go through the extreme song and dance of the PIP process and documented reprimands in order to provide counsel if litigated.
Yet I have met very few truly bad engineers in my life. Most of the "bad" ones were not bad in skills, but a bad match due to their willingness to die on one hill or another and complete refusal to work with others.
Yet, most of the interviews put way too little focus on the soft skills and way too much focus on the hard skills.
hard skills are difficult enough to try and assess; soft skills even harder. Most try with behavioural questions which have very little signal IMO. I'm a senior manager / director now so most interviews focus on softer skills. My strategy is to give very concrete examples to the "tell me about a time" style questions. Every other answer is easily forgettable.
In what companies are you speaking of? In Silicon Valley, we fire people very quickly as soon as notice it’s not a correctable problem. Hell, sometimes even when it is correctable.
> […] and then having a conversation about what is done, reasoning, etc.
Isn’t this where it would likely unravel?
The interviewer will know what the interesting parts of the exercise are, and ask the deep questions about them. Observe some more: do they know how to use an IDE, run their own program, cut through code to the parts that matter. Basically, can they do the things someone who wrote the code should trivially be able to do?
Since it was mentioned in a sibling comment: Even if the candidate used an LLM to write the code at home, I don’t care, so long as they ace the explanation part of the interview.
Agreed. It's one thing to ask the AI to solve the problem; it's another thing to be able to explain the way the problem was solved in real-time.
(Though you have to watch out for folks that are using the AI to answer your questions.)
In fact, I'm okay with people using AI to solve coding problems, as long as that is acceptable behavior at work as well. That should all be spelled out in the interview expectations.
> Though you have to watch out for folks that are using the AI to answer your questions.
Heh I do think that happened once (that I was aware of), but it was on a topic I knew a lot about, and it fell apart after layer one. Still, pretty lame, I’d much prefer a “I don’t know,” which I usually say if they start guessing.
We were worried about that as well. But we have found that most people are not doing well on our take home. If we get to the point that most people are crushing it, then we may need to think more about AI and take homes (maybe tweak the it with the explicit expectation that they may use AI, etc.)
They also need to be able to reason well about why they made the choices they did. Something useful when talking to them can be asking questions like "If X changed, how would that impact your design?". If they were reliant on AI for vibing (rather than just using it as a tool), then those can be more difficult questions to answer well.
If a person can use AI to complete an interview task, why don't you think they'd be able to use it to complete a work task? I think this thinking is flawed. 15 years ago the argument was, "they'll just use StackOverflow during their take home test." But then once we all got the job we'd check StackOverflow to help solve problems we come across. I don't think AI should be treated any differently, it's a tool to complete the job.
the point of the take-home is not to assess the answer, but use it as an anchor to discuss the how / why / what else? type questions. If someone used AI for just the results this is obvious. If they used AI to get the answers AND learn / understand what wa produced, that's probably the new reality
Why?
Because the vast majority of job interviews are with terrible candidates, even if the majority of candidates are excellent. This apparent paradox has a simple explanation: excellent candidates selectively apply to a few companies and get interviews/offers at almost all of them. On the other hand, terrible candidates are rejected at every step of the hiring process, and have to constantly reenter the interview pool.
Suppose 90% of candidates are excellent and 10% are terrible. If the excellent 90% only need to interview at one company, whereas the bad 10% need to interview at 20 companies, then only 0.9/(0.1*20+0.9)=31% of interviews will be with qualified candidates. To retierate: almost 70% of interviews will be with terrible candidates, even though 90% of people applying for jobs are excellent.
Because the cost of a bad hire is so consequential, the interview process is not designed to efficiently handle a minority of qualified candidates, but rather efficiently weed out a majority of horrible candidates. It is therefore a terrible process for the people actually qualified to pass it.
Why put the blame only on candidates? Interviewers are equally bad to interviewees. I have been to both sides of the table and can guarantee that 80% of interviewers would not be fit for my job or the process of hiring.
I don't think the parent comment meant to 'blame' the candidates; I read this as a statistical picture. Because of how the numbers work out, the market (as measured per-interview) is flooded with bad candidates. This does not disagree with the fact that companies are _also_ usually pretty bad at interviewing.
Same story, I think. Well-paid positions at sensible low drama companies are filled quickly, while companies with glaring issues may interview and make offers to dozens of candidates before finding one who accepts the offer. So as a candidate you also see a disproportionate number of bad interviews.
Your point is interesting but I don’t see how it answers the question.
“The majority of interviews being statistically rejections “is not obviously related to a need to make the interview process more demanding than the actual job - Testing for the role would also weed out bad candidates.
Here’s some alternatives:
- engineers testing for the idealized version of their job, rather than the realities of it - “a true engineer must know how to balance a binary tree! Nevermind that I spend my time dealing with support tickets regarding a null pointer that slipped by in a code review”.
- companies using long processes as negotiation tactics - “you went through two months of trials so you’re less likely to reject our offer now and start over”
- design by committee - “every interested party and team must give an approval in their own step”
- interview as marketing - “see? We deal with tough challenges, this is an interesting place to work at”.
As someone interviewing for the first time in a long time I can tell you most assuredly that there are a disproportionate number of awful interviewers too.
Of the interviews I've had I would say about 3/4 have tried to catch you out with some inane gotcha that you would never see in the wild or have a very specific solution in mind without exploring or discussing. Sometimes both.
I can attest. I've personally been shut out of two positions because I didn't say the "magic word" that the interviewer was looking for. In one instance, I kept using laymen's terms to describe an optical engineering process, because I knew they weren't optical engineers (far from it). They wanted me to say the precise technical phrase, but wouldn't explicitly ask that; they just kept asking questions around that phrase.
In another job where I was hired, I was tortured for over 10 minutes as one interviewer tried to get me to say the magic word. Fortunately, his manager was present, and shut down the questioning at that point. I never did learn what word the subordinate was trying to get me to say, but I was fully qualified for the job, and hired.
My more recent interviews have been fine, but for awhile I had several that were (in the words of xkcd) "communicating badly then acting smug when you're misunderstood".
We have to acknowledge that most companies are horrible at recognizing a bad hire and then appropriately handling the situation. A lot of companies profess to "hire fast, fire fast" but very few actually do. Until that changes, the cost of a "bad hire" will continue to be disproportionately high.
I think this may be an example of Simpson’s Paradox
https://en.m.wikipedia.org/wiki/Simpson's_paradox
Ah, the "market for lemons" argument: https://www.jstor.org/stable/1879431
> Because the cost of a bad hire is so consequential,
This is stated all the time and I feel it is true. But is there any way to make it less consequential? That was my main argument for contract-to-hire (though I know there are downsides to that approach).
Are there any other ways to make hiring less risky?
> Are there any other ways to make hiring less risky?
Professional licensing.
Many other fields require professional licenses. I don't understand there's so much opposition in our field.
(Ok, I do understand.) In general, licensing has some risks:
The lemons will get excluded from the field. (Which is kind of the point.)
Or, the lemons will decide what the criteria for a professional license is; which turns it into a BS hurdle.
---
That being said, the article gets closer to the point of what a professional license is for: "An interview is like running 100m and a job is like a 10k.". If the license is more like running a 10k, then interviewers can rely on it to do a better screening than they could ever hope to do.
The issue with professional licensing, is that it's very, very specific.
In things like civil engineering, there's usually mandated context. You have to work within certain parameters, so it's not too difficult to test with real-world criteria.
With software engineering, it's all over the place. In fact, one of the most exciting things that I used to look for, in potential candidates, was people who were not bogged down with dogma, and would bring alternative viewpoints to the team.
Since anyone coming into my team would require a ton of training; regardless of their seniority, I always had a nice, long on-ramp, in which I could evaluate people.
There can be a halfway solution that helps: A professional software engineering license that signifies an extremely basic, barebones level of skill. Companies would still need to interview candidates, but they wouldn't have to do FizzBuzz or "write a for loop" types of interviews with candidates who literally cannot write code at all. It wouldn't guarantee the person was an expert in inverting binary trees, but it would at least represent a minimal knowledge and skill bar that one would expect any software engineer to meet.
One of my interview questions used to involve putting up a parameterized SQL statement on the whiteboard, (select userid, email from users where userid=?), give the candidate instructions on how to use a data reader, and then expect them to construct a "user" object.
I did this because I was part of a project that failed because the leads did not know how to work with a database. They thought the ORM would just magically load objects. Without understanding the basic limitations of data readers, they painted themselves into a corner.
Licensing basically needs to filter this kind of thing. IE, you don't get a license to work with a SQL database unless you can work with a data reader.
---
The irony was that we very rarely touched SQL, but the question was one of the greatest filters out there! It showed who understood basic concepts, who was just BSing their way though, and who would get stuck in high-level abstractions.
It really, really sucks, that we have to be here.
Maybe have online application pages, that can’t be submitted, unless you fizzbuzz. :(
One additional downside to professional licensing is that it can be time consuming and costly. For example, to bring a new physician on board to a practice they must be:
* Licensed (allowed to practice in that state)
* Credentialed (degrees and experience verified)
* Enrolled (able to be imbursed via insurance)
* Priviledged (authorized to perform certain tasks/roles)
This could take weeks or months per physician and there is usually an entire team (Medical Staff Office/MSO) dedicated to the work.
> This could take weeks or months per physician and there is usually an entire team
That seems like a better alternative to what we have now! Some companies search forever for the "perfect" candidate, or otherwise have to sift through so much SPAM resumes that they are already taking "weeks or months."
Kind of like how the CFA used to be in finance.
In my experience, the hiring managers with the best track record have these in common, which mostly boil down to the hiring teams doing their homework:
(1) The expectations for the position are clearly defined (2) The hiring team members coordinate on questions and expected responses, and they are consistent in interviews. (3) The hiring team members know how to spot potential issues (e.g., excessive bad-mouthing of previous employers, etc.) (4) The hiring teams effectively leverage their networks for references. Ideally, there are not-too-distant trusted relationships between the candidate and the hiring team. In the absence of this, references are followed up on carefully (this has become an art form in modern times.)
These reduce the risk of someone slipping through the cracks. Hiring teams also get better with experience, so any mistakes should be carefully analyzed and corrective actions incorporated into the hiring process.
Trial periods. Asking employees to do less in their first months.
Maybe not great mitigations, but that’s what I could come up with
After interviews, you hire Alice but Bob was a close second. Bob goes to a competitor. Alice doesn’t work out. A double whammy. Why not immediately try to get Bob? I’ve only seen this a few time.
Relatedly, if you hire Alice and she works out, but in months or a year, you hire for the same role, why not reach out to Bob again?
You invest all that time into interviewing Bob, but then if they don't get the offer, you never reach out to them again. I don't get it.
I don't think I've ever seen this done well.
In the US at least it seems common to review performance and put certain benefits on hold for 90 days to see if it's working out. This wouldn't mitigate the costs of the initial hiring or the opportunity cost of not selecting another candidate, though.
The 'cost of a bad hire' is received wisdom that needs to go away. The first order effects of your team's time investment are easy to see and make good content for your engineering leadership blog when you're aiming for promotion. The second order effects are what get debated in threads like this ad infinitum.
Paradoxically, a higher bar for hiring increases these consequences for everyone. A bad hire is only consequential in the first place because hiring managers are slow to cut them loose. Managers are slow to cut loose because they are morally culpable for the consequences to the individual they hired. When a manager extends an offer, they are accepting some responsibility for a significant change in a person's life. It's very difficult to walk that back when it's a bad fit, knowing that hiring is a slow process and every other company out there is scared of making a bad choice. But at the end of the day, interviews are an approximation of the candidate/company fit in what is ultimately a matching problem. More attempts make for better matches. Companies and candidates both would be better served by being faster to hire and faster cut loose.
This is closely related to the claim "We only hire the top 1%". Yes, but when your competitors reject a candidate they don't just stop existing, eventually they choose you.
Yes... Joel Spolsky puts it a bit uncharitably here [1]:
> I’m exaggerating a lot, but the point is, when you select 1 out of 200 applicants, the other 199 don’t give up and go into plumbing (although I wish they would… plumbers are impossible to find). They apply again somewhere else, and contribute to some other employer’s self-delusions about how selective they are.
----
[1] https://www.joelonsoftware.com/2005/01/
I mean this is certainly an effect, but largely outdated reasoning. Every tech company you can think of copied verbatim a Google-style interview process out of complete reflex without understanding largely why. Now you discover that your candidate, who can invert a tree just fine, and can regurgitate quicksort, is totally useless at delivering software, managing stakeholders, or understanding the business logic.
Nowadays it's more like the excellent 90% need to interview at 20 companies and the terrible 10% need to interview at yet more
best explanation i’ve read so far!
If I remember correctly, there's an ancient article by Joel Spolsky about this.
That great candidates are not out there doing blind interviews, only those of us who are average are interviewing and going through hoops. Engineers below average are almost constantly interviewing.
Yup, back in 2005 [0]! It's far from a new idea.
[0] https://www.joelonsoftware.com/2005/01/
It’s rather older than this. Any argument about “the market is swamped by bad X in circulation and good X are much rarer in the market than you would naively expect” (software developers, online dating, used cars, debased currency…) is just a version of Gresham’s Law, which is 500 years old: https://en.m.wikipedia.org/wiki/Gresham%27s_law
> Because the vast majority of job interviews are with terrible candidates, even if the majority of candidates are excellent. This apparent paradox has a simple explanation: excellent candidates selectively apply to a few companies and get interviews/offers at almost all of them. On the other hand, terrible candidates are rejected at every step of the hiring process, and have to constantly reenter the interview pool.
Do you have actual data showing this? Or is this just your intuition?
Because if it's the latter, my intuition is pretty different.
No one is an "excellent candidate" for every position, and many people who are "excellent candidates" for a given position might not even recognize that excellence in themselves. Therefore, they're not going to be only applying for positions at the very best companies; they're going to be applying for any position they think they have a chance at, assuming they think they can actually be OK with the job (eg, they might not want to apply for a job in adtech if they are personally repulsed by the ethics of surveillance capitalism).
Additionally, your scenario sounds like it paints the candidate pool for jobs in general as a bimodal distribution, with one peak of "terrible candidates" and one peak of "excellent candidates", with very little in the middle. My intuition says that it's much more likely to resemble a normal distribution.
No; what's much, much more likely is that most people are decent candidates for many jobs in their field, and excellent candidates for a few, but their chances of actually getting the opportunity to apply to those few (between the position being open and them searching at the same time, and them finding out that it's open) are slim, so they have to apply for a great many of the positions that they're only decent candidates for. That means that they'll try many times before finding something. This can lead to a lot of frustration and even desperation, creating a willingness to engage in some shady techniques to actually get a human to talk to you and recognize your value.
Then there are a few people who are, indeed, nothing but shady techniques, and they are likely to flood all channels with as many AI-generated or otherwise low-quality applications as they can manage.
So no; even if most applications any position sees are terrible applications, most interviews are likely to be with decent-but-not-excellent candidates, and most people are still going to have to interview with a few or even tens of companies before they are actually offered a position.
In my experience, technical interviews are not really useful past the very basic "Can this candidate actually write a conditional and has the slightest clue about programming?". Ability to solve hard leetcode-like problems under time pressure in a stressful environment doesn't meaningfully translate to "will be a great contributor to the team work on the kind of problem we have".
Our best hires are nearly always coming from the network of a team member or people we contracted with and decided to hire full time.
Most of my time in interview nowadays is spent understanding what the candidate has done before, explaining to them what we do and asking open questions to see how they would approach our issues and how they link them to their experience. If it seems to fit, we hire. My country standard contract offers a fairly long probation period for new hire and we don't hesite about parting with people when it's not working after a quarter. We are very explicit about this policy.
> Our best hires are nearly always coming from the network of a team member or people we contracted with and decided to hire full time.
I love this approach.
However, at least in the USA, there are substantial costs to candidates for this approach:
- no health care while contracting (unless the candidate pays for it)
- being a contractor is a different level of risk than moving from FTE to FTE
- if the employer decides it isn't a fit, candidate has to find another contractor
Do you have any great candidates who approach you and then, finding out your approach, pass?
> Do you have any great candidates who approach you and then, finding out your approach, pass?
We never generalised the contractors to employee thing. It's just that we hire contractors fairly often to fill in temporary needs and we generally extend offer for full time employment to the ones we would like to keep with us when we can. They generally say no because they like being contractors.
For candidate applying, I have yet to see one explicitely refuse an offer because they know we don't always keep going after the probation period. Then again, the standard probation period for engineers in my country is 3 months which can be renewed once so they would get the same offer anywhere.
But I have never seen someone being surprised when we parted after the 3 months. If someone told me they were during our offboarding meeting, I would take it as us having done something seriously wrong during our onboarding.
To be honest, we don't have that many hirings which end up not working and of them I can only remember one being due to an actual lack of skill from someone who clearly padded their resume yet managed to go through the interview. Some people needed a bit more time than others to reach the level of delivery we expect but that's ok. We don't always need rock stars, just professional who can deliver consistently and are open to learning new things. When we need specialists and don't have someone sufficently skilled internally, we contract. Working together with a specialist tends to raise the level of the team as a whole.
Most what I consider our true hiring failures have come from a mismatch between what the person expected and what the job actually is. That's why I now take time to ensure the person I'm interviewing actually has a good idea of what we are doing.
I find problem-solving questions always pretty good at weeding out candidates. Give them a real world problem with a fictional situation from their future job and just ask how they would tackle the problem.
Invert a binary tree, find the optimal path in an n-dimensional space of real numbers given by a function, design a horizontally scalable build system, get grilled by at least two people on your leadership skills... then, if you're extremely lucky, get hired to configure the button to be blue and have round corners. And get laid off in 9 months.
(Not saying this happened to me but it's a common story I've heard in the last few years)
I agree with the author that it is hard to assess someone's skill if you have a list of 100 people to interview and you know nothing about them. The bigger the stack of applications the easier it is to treat them like data and not people.
I know plenty of software developers who write, maintain, or contribute to OSS libraries; they write blog posts, give talks at conferences/meetups, and make videos in their spare time (or sometimes as part of their work). I've rarely walked into an interview where the hiring manager or the technical interviewer hadn't just read my name off my resume as I joined the meeting. Get to know the candidate's work before assuming they know nothing!
Maybe people involved in the hiring process should be given more time to properly research a candidate. Relying on ATS' and putting the burden on applicant's to do the work of proving themselves is causing a lot of folks to burnout just trying to get a job.
Jm2c but interviews tell you absolutely none, nothing, about what kind of a professional the candidate is.
I have no clue whether he'll care and help or pretend to work and drag everybody else down.
There's a huge number of incredibly capable developers who could pass any interview but then spend days playing video games and sabotaging projects and teams.
I really don't believe in technical interviews, I'd rather base the relationship on trust, if you tell me you're good/experienced at X I trust you to be. If it was bs you'll be shown the door with ease.
Instead many companies make it insanely hard to get you hired, but also incredibly hard to cut you out even if you're impact is a very net negative.
This works very well for contractors, less so for full-time employees. You can't just fire somebody on a whim, at least not for free, not even in the US, let alone in most of Europe.
To be clear, I'm not saying worker protections are bad, just that if firing is much more expensive than hiring, you can't really afford to hire any warm body that walks through the door. These days everyone and their mother is in CS, there are many more talented people than ever, but also more duds than ever.
Yes, you can even in Europe where you have high protections like in Italy, there's a probation period. Standard is 3 months, higher seniority can be even 6.
It's also the reason why e.g. Italians don't change jobs for 10/20% higher salary, you don't want to go through probation.
> firing is much more expensive than hiring
So... The solution is putting your most senior people doing week-long interview rounds for each candidate?
IME: the only valuable signal comes from direct personal referals. If you have someone who you think is good, and they recommend someone who they say is good from a previous engagement, odds are it will work out. There's a transitive, holistic measure in play; they're not going to destroy their reputation by recommending a weak player or even a strong jerk. The problem here is scale (you quickly milk direct networks dry) but nothing else seems to work well.
> firing is much more expensive than hiring
It's the reason why most jobs, even in Europe, have a probation period. During this oeriod firing is inexpensive. In my current employment the probation period was 6 months.
If in 6 months you still can't figure out if a hire was good or not, interviewing won't save you.
> Jm2c but interviews tell you absolutely none, nothing, about what kind of a professional the candidate is.
With the caveat that I have not been on the hiring side of things myself, my feeling about this is that it can—if a) the person you're interviewing is acting fully in good faith, b) they're not the type of person who gets so nervous about interviews that it skews their whole personality temporarily, and c) your interview process is actually decently-designed (I don't think leetcode or any type of "gotcha"-questions will give you a good picture of how professional a candidate is, nor will an interview for any kind of technical position designed entirely by nontechnical people with no domain input).
The big, big problem is the genuinely small proportion of people who come into a job interview in bad faith, seeking not to demonstrate but to hide their true skill and personality, because (rightly or wrongly) they believe it will hinder their chances of a good position.
The alternative I prefer isn't any of the given ones, but a scaled interview. I start with something that you can solve with a Python one-liner, if you know what you're doing, but if you solve that instantly I've got a series of questions that will scale up until we're eventually doing the task using untrusted user input, outputting into a security-sensitive environment, under heavy load that we want to be cheap, in a highly-available environment, etc. etc. I also tell the applicant up front that this is the plan, because if you don't do that, it feels like you're jerking them around by constantly changing the requirements. I've also got a number of directions it can go depending on the applicant, and depending on what they tell me about their past experience.
The other nice thing about this is your interviewee generally doesn't leave feeling like a failure; it's not like I have three questions and you can get them all wrong. Unfortunately there are some people who end up spinning on the very easiest question for the entire session, and, uh, well... I can only do so much, if you really don't know anything about programming at all. This is at least the exception, though.
I have not had to do an interview in the age of practical AI yet, though. In person I don't think I'd have to change much, I've always interviewed with a policy of "I'm not worried about whether the string split command takes its parameters in this or that order, I just want to know you know it exists" and I can basically serve as an AI in the same way I was already being the API reference. Remotely, I'm not sure what I'd do yet.
I do like your approach more than many alternatives but there are still lots of challenges. Two big issues: 1. this is not typically how even a confident, prepared, solid developer approaches coding problems, so they can be poorly prepared for this style of interview. The result is you don't get an accurate representation of the individual. 2. it takes an awful lot of the interviewer's time to prepare, even if you can scale to multiple candidates with much of the same work. You need a very strong technical resource to deliver this style of interview, and they could probably be doing higher value work.
I've hired a lot of the past ~5 years and my take away: it sucks for everyone - on both sides - except the early HR-led functions, and they don't add any value or much help. They may actually make most of the challenges much worse.
"this is not typically how even a confident, prepared, solid developer approaches coding problems, so they can be poorly prepared for this style of interview. The result is you don't get an accurate representation of the individual."
It may not be how you approach coding problems but it tends to accurately reflect how people program in real life, so I don't think it's all that much of a stretch to ask people to show it in an interview. Nobody gets an assignment to write a scalable, testable, blah blah blah 100 pages of requirements project and just sits down and writes it linearly from start to finish. Everything is incrementally developed.
"it takes an awful lot of the interviewer's time to prepare,"
Perhaps in general, but not for me past the first couple of times. However, that's because I do strongly agree with...
"You need a very strong technical resource to deliver this style of interview"
Yes, definitely.
However I don't believe there's a way around this. I don't believe you can get high-quality results for technical hiring from any possible "interview" you can hand to a non-technical programmer.
Moreover I don't even know why people think this is or should be possible. Other than the popular "I want it therefore it should be possible" argument, which the universe doesn't particularly respect. I wouldn't expect that even armed with a prepared formal interview that I could interview an industrial chemist, oil rig engineer, or a plumber. There still seems to be a certain amount of residual contempt for programming in some circles and that includes the idea that a non-programmer should be expected to be able to interview a programmer at anything more than a super-basic level. No, programming is at this point as technical as any other career out there. It really wasn't 30 years ago, but it is now, where the basic standard for the simplest possible project now encompasses a suite of technologies and competencies just to get started and the complexity goes up from there.
Also, while a senior engineer perhaps shouldn't be doing full time interviewing, interviewing is extremely high value work. It determines the course of the entire team for years at a time. The interviews I did years ago still determine the course of my day-to-day activity. Huge impact.
That is a really neat interview format, the lightning round of varying themes of common tasks! This right here proves to me you are a good interviewer:
> I'm not worried about whether the string split command takes its parameters in this or that order, I just want to know you know it exists
I’ve run quite a few “can you write code” interviews in the age of practical AI, and I don’t know if I’ve been lucky, am good at breaking through nonsense, or if internet claims are exaggerated, but I can hardly tell the difference between now and the before times. You get someone on a call, you explain a problem, you see how they approach it, you probe along the way. I don’t work for a giant FAANG-like, maybe that’s part of it.
Valid points, but I think the most obvious reason is that (at least the recognizable) employers are swamped with applications that all look great on paper, which likely got much worse with the rise of good LLMs. Compared with similar high paying carreers, like medicine (multi-year residency) or high finance (solving math and probability problems on the spot), the hiring process for software engineers isn't especially gruesome.
"multi-year residency" in medicine is part of the education.
We don't have that for software development.
I'd say that the problem is that the diplomas (degree certificate, ...) are useless when hiring software developers.
A doctor who has a diploma is much more likely to be a useful/good doctor than a person with a "computer science" diploma will be a good developer.
> A doctor who has a diploma is much more likely to be a useful/good doctor than a person with a "computer science" diploma will be a good developer.
A few things to consider:
What we practice is "software engineering." Computer science is closely related to software engineering; but not the same thing. (It's like the difference between a degree in physics vs mechanical engineering.)
Doctors still have to be board certified, which requires self-study of topics that aren't taught in class. Some people do get their medical diploma and fail their board certifications, in which case they can't practice.
I should add: One of things, for doctors, that board certification does is make sure that doctors know topics that aren't covered in school.
My computer science degree overlooked a lot of topics that are very important when developing an industrial strength application; specifically, an application that is not an academic exercise, which gets back to:
> A doctor who has a diploma is much more likely to be a useful/good doctor than a person with a "computer science" diploma will be a good developer.
If we, as an industry, had something similar to doctors getting board certified, we could put pressure on ourselves (and academic institutions) to make sure that ample opportunity is provided in schools to learn how to be a good developer.
Back when I started, I think maybe half of the people I worked with had a relevant degree at most. Even that might be an overestimate.
Back then, CS degree programs didn't even teach higher level languages than C/++ as they were thought to change too quickly. Whatever you learned during your four years wouldn't apply after you graduated. Instead, the programs focused on the low level implementation details with the theory that was where the engineering and science were.
Now, the same school I went to has courses and tracks for web technologies, so who knows.
Companies keep ignoring the historical interview point. I have been to a few occasions when companies needed the exact thing i built in the past (e.g. migration to clickhouse), but chose to put me into a random take home related to a different technology (e.g. some bigquery assignment) and eventually reject me. Go off script and ask me details about the project which might solve your hands, why do i need to talk about something else?
quick answer: this takes time & energy; the interviewing company is willing to miss you to save this effort. Plus the interviewers are not likely to be good at these tasks, even if they're solid developers; it's a skill they have relatively little practice and training.
I understand this for a dry candidate pool. I have been there as a hiring manager, you need some signal with take home or live interviewing. But on the rare opportunity that you find someone who is willing to talk in details about a similar project you want to do, you skip the pipeline and if he indeed built it, you hire him.
Interviews allow just a little time for strangers to attempt to sync up and get a sense of each other.
Actual jobs offer much more time for people to learn to sync and communicate.
Also, on both sides of the interview table, people have varying strengths when it comes to short and long term communication skills. Plenty of interviewers are not good at interviewing, just as plenty of candidates are not good at their side of the process.
In short, interviews are a very poor approach to choosing who belongs long term.
Ultimately, regardless of the interview process used, every job is ultimately a long term “try and see” interview. You only know how someone fits by trying them for a while.
Good post. Thanks for that.
I tended to do the "Historical Interview" one. In my case, it worked well.
I think take-home tests are probably damn near worthless, these days. People will just feed the assignment into an LLM, and return the results.
I am not a fan of LeetCode, because I don't want to waste time, studying stuff that won't have any relevance to what needs to get done. I like to keep my dance card full, and wasting precious time to make someone else happy, isn't my idea of a good time.
That said, I understand why they are there. I just feel that they aren't really something that I'd use to judge senior-level talent; which was what I used to hire.
> Asking a candidate to solve some problem which lets a candidate work in a slightly less high stakes environment. but requires candidates to do extra work, taking longer than just the interview.
I refuse a lot of these. I've put a lot of time into many of them, just to be ghosted.
Typically, when the take-home comes with a laundry list of frameworks and 3rd party libraries, I walk away. Coming up to speed with all of them is too time consuming for the probability of getting ghosted.
It takes a lot of work on the interviewers side to come up with a good take-home. It requires discipline to reject the temptation to just throw in a laundry list of requirements.
I know it is hearsay, but it seems many times interviews try to get free work from candidates. Here, have a look at this real problem, see if you can solve it in whatever the interview time frame is.
When there are too many candidates and not so many jobs, and absolutely no regulation, employers are free to exploit applicants without consequences. Employers will do this even when there is no actual job being offered, so they can use the availability of applicants to apply downward pressure on the salaries of the workers they already have.
I have never seen this from either side. Think about it from the hiring side: how good of a solution could be produced by someone with zero context and experience in the business, domain and internal environment? This would be like less efficient "organic" vibe coding.
We should establish a convention that # noqa cannot be removed from the top of my code sample.
Interviews have been extensively studied by phd, figuring out what works and what doesn't. I have yet to see anyone write a blog or comment that shows even an awareness that this research exists, much less what it in it. (I know this research exists, but I'll admit to not knowing how to find it. My company insists the interview process they make me follow is based on it)
Cargo culting, and the fear of bad hires. Everyone wants "the best engineers". Probably 1% of tech firms work with deep tech, where boundaries are pushed, and every percentage of improvement or degradation can result in non-trivial gains or losses. The rest can do just fine with normal engineers, and do interviews like the rest of the professional world.
The 100m / 10k analogy is fantastic, but the reality is even worse. Let’s say you expect a candidate to stay for average five years. That’s about 1200 working days. If you get to interview them for 1 day, then that day is 8.3m to the 10k. You get to watch them run for 8.3m and decide how well they can run the 10k.
You're not even watching them for the skills required on the job either. Most jobs aren't even about running at all.
Basic, boring, line-of-business web application shops are asking candidates to implement Boyer-Moore and k-d trees. When the applicant is going to be hired to mostly update JSX templates and figure out why the customer is frustrated. It's like every business needs Olympic athletes when they're running a hot-dog stand.
It's one thing if your business is writing a RTOS for a custom platform you develop. Asking a candidate to implement a slab allocator might be a fine exercise. It fits the work being done.
Businesses need to get off their high horse and look in the mirror. Most of them are doing mediocre things that need average, good-natured people who are willing to work.
I think there's also another element that the author is missing and that is likability. I've never experienced this, but I've heard about people being put through grueling interview processes merely to weed them out because they were deemed "not a good culture fit".
I think at least half of all interviews are a popularity contest, and the other half are your qualifications.
Not a fan on hiring based on popularity, but I do wish companies spent more time on actual culture fit and whether or not the person they are hiring is a raging asshole. I've worked with my share of "brilliant jerks" and they're just awful to be around. Even if they are John Carmack level of coding, it's not worth it--they are a net negative because of things like poor attitude, lack of communication skill, arrogance, my-way-or-the-highway stance on everything... I wish that filtering these people out was 50% of my company's interviews!
And I can tell from experience that it's quite easy to get rid of "brilliant jerks" during the recruitement process.
The key is to make them stop the process, not you; so you won't have to deal with HR or management telling you that they are brilliant and you should not reject them just because "you" don't like their behavior.
Scaring jerks out is quite easy: just be excessively nice during the interview. Tell them how much time and energy everybody take caring for each other's well-being and feelings. Tell them that in the team you have lunch together every single day. And you do stuff together after works. And during the weekends.
They'll just run.
Sure, you're lying because you're working in a regular team where people are simply respectful professionnals, but they don't need to know that. You'll do everybody's a favour. The jerk included.
I don't think "popularity contest" is accurate, but cultural fit is both a real - and important - component AND a risky opportunity to apply massive personal bias. Lots of variability here; I have a coworker who asked the exact questions to all candidates to avoid this, while I can't stop going off script. I know my risk is higher but I also find better candidates (while very likely passing on very good candidates for the wrong reasons).
Is there any other industry with such a disconnect between education and job where the candidates need to do another exam when they try to join the job?
A major issue the author missed is it is much harder to fire a non-performer in the tech industry today.
It takes 2 quarters (ie. 6 months) to go from recognizing a problem employee to firing said employee.
This makes the risk of hiring the wrong candidate significant as a hiring manager, because a bad hire reflects badly on you and eats up your budget thus preventing a backfill.
On top of that, firing individual employees can lead to litigation risk (even if frivolous), thus requiring hiring managers to go through the extreme song and dance of the PIP process and documented reprimands in order to provide counsel if litigated.
Yet I have met very few truly bad engineers in my life. Most of the "bad" ones were not bad in skills, but a bad match due to their willingness to die on one hill or another and complete refusal to work with others.
Yet, most of the interviews put way too little focus on the soft skills and way too much focus on the hard skills.
hard skills are difficult enough to try and assess; soft skills even harder. Most try with behavioural questions which have very little signal IMO. I'm a senior manager / director now so most interviews focus on softer skills. My strategy is to give very concrete examples to the "tell me about a time" style questions. Every other answer is easily forgettable.
In what companies are you speaking of? In Silicon Valley, we fire people very quickly as soon as notice it’s not a correctable problem. Hell, sometimes even when it is correctable.
This is going to be the new reality of an AGI future.
I prefer take home, and then having a conversation about what is done, reasoning, etc.
It feels like the age of AI has made take home coding exercises obsolete. I still have not found a good replacement.
> […] and then having a conversation about what is done, reasoning, etc.
Isn’t this where it would likely unravel?
The interviewer will know what the interesting parts of the exercise are, and ask the deep questions about them. Observe some more: do they know how to use an IDE, run their own program, cut through code to the parts that matter. Basically, can they do the things someone who wrote the code should trivially be able to do?
Since it was mentioned in a sibling comment: Even if the candidate used an LLM to write the code at home, I don’t care, so long as they ace the explanation part of the interview.
Agreed. It's one thing to ask the AI to solve the problem; it's another thing to be able to explain the way the problem was solved in real-time.
(Though you have to watch out for folks that are using the AI to answer your questions.)
In fact, I'm okay with people using AI to solve coding problems, as long as that is acceptable behavior at work as well. That should all be spelled out in the interview expectations.
> Though you have to watch out for folks that are using the AI to answer your questions.
Heh I do think that happened once (that I was aware of), but it was on a topic I knew a lot about, and it fell apart after layer one. Still, pretty lame, I’d much prefer a “I don’t know,” which I usually say if they start guessing.
We were worried about that as well. But we have found that most people are not doing well on our take home. If we get to the point that most people are crushing it, then we may need to think more about AI and take homes (maybe tweak the it with the explicit expectation that they may use AI, etc.)
They also need to be able to reason well about why they made the choices they did. Something useful when talking to them can be asking questions like "If X changed, how would that impact your design?". If they were reliant on AI for vibing (rather than just using it as a tool), then those can be more difficult questions to answer well.
If a person can use AI to complete an interview task, why don't you think they'd be able to use it to complete a work task? I think this thinking is flawed. 15 years ago the argument was, "they'll just use StackOverflow during their take home test." But then once we all got the job we'd check StackOverflow to help solve problems we come across. I don't think AI should be treated any differently, it's a tool to complete the job.
the point of the take-home is not to assess the answer, but use it as an anchor to discuss the how / why / what else? type questions. If someone used AI for just the results this is obvious. If they used AI to get the answers AND learn / understand what wa produced, that's probably the new reality
In office whiteboard interview
[dead]