The problem is my job is more dev-lead *seeing around corners and pushing the ball forwards) stuff, and architect stuff - vs. say coding a framework from scratch or anything that needs intensive algorithms.

But you can't easily test for that stuff. So they say "what's your best language" and expect you to be a guru. Which is probably fair and probably a deficiency by me that I'm not a JS guru.

But it's absolutely not a job requirement for what I do or the jobs I interviewed for. What was a requirement for some of them was deep node experience. But nobody at the companies had that either. So they didn't know what to test for and didn't really understand my OS framework. So they just reverted back to involved JS scope questions and stuff like the hold water thing.

Somehow the rest of the job market gets by ok with just interviewing people and checking their references. Not sure why programming has to be so special that 90% of it comes down to a timed test in front of a panel. And again - I get that for FAANGs with a million applicants it might make sense. It does not make sense for a startup imo.

I totally get that something like fizz buzz or a little more makes sense. You want to make sure they're not just completely incompetent. But I take issue with the idea that actual rock stars vs. whiteboard rock stars is going to be tightly correlated. A lot of companies seem to agree with that too - that it's just a check of basic competence but hard to read in much more. Of course some people have been buried deep in some technology for so long - they except basic competence to look like the stuff they've known for so long they forgot when they learned it.

The one job that I almost got which had to let go the previous dev for "not enough node experience" apparently had to let go the guy they hired instead of me for the same reason. They were asking about me 6 months after I took this job. Lol.

But I didn't wow them on the whiteboards (I didn't flub them either) and I told them I wasn't into gaming - lol. They asked zero node questions. It probably wouldn't have been the best fit but it was a lot closer to my house, and they let you wear shorts and work from home whenever you want.

it's a bummer because I consider that node framework my crowning achievement as a developer. It was so perfect for what we needed to do (and perfect for any REST-IN/REST-OUT node orchestration layer really). It's right there for anyone to inspect. And it counts for nothing apparently.

Btw anyone can feel free to take a stab at the water problem. It's kinda gnarly. I think we already did it in this thread though.

Or if you have what you think are good whiteboard questions - feel free to post.

Interviewing is a Noisy Prediction Problem

To me it was informative article. It cites studies and gives his views based on extensive interviewing of developers.

His overriding theme on interviewing:

Quote:

I have done roughly 2,000 interviews in my life. When I started recruiting, I had so much confidence in my ability to assess people. Let me just throw a couple of algorithm questions at a candidate and then I’ll tell you if they are good or not!

Over time I’ve come to the (slightly disappointing) realization that knowing who’s going to be good at their job is an extremely hard problem. The correlation between who did really well in the interview process and who performs really well at work is really weak.

Confronted by this observation I’ve started thinking about this process as inherent noise reduction problem. Interviewing should be thought of as information gathering. You should consciously design the process to be the most predictive of future job performance. Given that you have limited time to measure, you need spend your time measuring things that have high signal-to-noise ratio and things that have low correlation with each other.
....

I’m not a scholar and I can’t claim to have done an extensive literature review, but here’s my biased summary:

Intelligence tests seem to be the strongest predictors of job performance (although possibly offensive, and probably illegal in the US)
… followed by structured interviews
… followed by unstructured interviews (which have very little predictive power).
Picking the measurements that matter the most
Let’s focus for a second on what measurements matter the most. Since we can’t measure actual job performance, we need to measure other skills. Those measurements will be highly noisy. So we need to find measurements that (a) are decent proxies (b) have as high signal-to-noise ratio.

On solving problems on a whiteboard:

Quote:

I don’t think algorithms on a whiteboard are great, but mostly because they take so much time. Asking a candidate to implement Levenshtein distance can take up a full hour, and again, the opportunity cost matters. Quick algorithm questions (max 5-10 min) can be fine, although another concern I have with algorithm questions is that junior CS grads score well for no other reason than having their algo class fresh in mind.

I think it is fair to state that suzzer has the opinion that often the interview process weeds out developers that are capable of performing well. It is really bad when incompetent candidates get selected and competent candidates are rejected. It may be ok if the process is such that competent candidates are selected for the most part but other competent candidates are rejected for whatever reason with some of the reasons being irrelevant as to job performance.

You know what is highly correlated to performance? Evaluating someone on a short contract.

I get that the *best* devs don't want to do that. But it might be a great way to pick up diamonds in the rough who don't whiteboard well - which a smaller company could use as an edge (and many do).

Or maybe just some super restrictive probation thing where you let candidates go if they aren't what you expect. In theory this is in place at most places. But it's only usually used if the candidate is grossly incompetent or threatens people or doesn't show up to work or something.

One reason is bosses don't want to admit they made a huge mistake. Even a horrible hire generally sticks around for 1-2 years until they get chances to fail on multiple projects.

For either of those you have to be in a position where you can hire say up to 3 people, but you're happy if 1 of our 3 hires works out. Which I get isn't always the case. For the FAANGs though it would be no problem. I guess you'd always have a two-tiered system though where the contract-to-hire people were looked down on.

my Scrum Master told us that we needed to "self organize around being more cross functional to improve our velocity hitting objectives and key results"

man I think youre one of the best posters but this analogy is legit awful. the problem with whiteboarding is that it doesnt in any way mimic what you are paid to do. the combine does at least show off athleticism.

Did he then publish an article or write a book about how his brilliant innovate approach increased efficiency by 37.698% in key objective measurables, and then leverage that publication to jump a pay grade to another job at a better company?

Well the argument is that whiteboarding at least takes smarts, which is needed. But we're talking pro sports for millions of dollars vs. a normal job. It's completely standard to expect someone to prepare for 3 months for a shot at that. At a regular job? You're filtering for people who are already good at the tests, or have a lot of time on their hands. Which is kinda the point.

I'd like to see rich white dudes forced to jump through dehumanizing whiteboard monkey hoops lol. Hey so we really like you as our new CFO. We're just gonna need you to come in for one day of special CFO financial tests on a whiteboard while we stare at you. I mean you might not use any of this stuff in your regular job - but it should show us your aptitude for being a good CFO. You may need to study up a bit. Cool?

There's also a component of being able to think on your feet under the spotlight in a big moment that exists in pro sports - which the combine simulates as does whiteboard. But that type of pressure just doesn't exist in programming. Yes there is pressure in programming. But it's not instantaneous. It's grindy. You have more than a few minutes to stop and think about what you're doing.

When I used to do standup - my worst trait was not being able to think of snappy spontaneous stuff in the moment for hecklers or whatever. Some people are just naturally great at that. I can't drown out all the other voices to hear the creative one with the right answer coming through. Oh but in the middle of the night the best comeback ever will come to me. I'd have gotten better at it over time, but it's just not something I'm good at. Hence I'm much more excited about the book I'm writing where I have time to really get it right. Programming is a lot more like writing than standup.

Btw, I did a little bit of research and it looks like the Netflix culture interview is what most other tech companies call a "behavioral" interview. It's a bad name but this has nothing to do with what most people think of culture fit and everything to do with how you compare against the stated principles/values/etc and how well you can map your past behavior in work situations to those. I don't particularly care for this type of interview, because it's not super hard to game if you know how the interview goes and it can be difficult if you're being honest and are not used to the typical workplace culture of tech companies (in the sense of how decisions are made, disagreements are handled, how responsibilities are divided and so on). I think some startups tend to do a real "culture fit" interview in the sense of who they want to hang out with, but that, as far as I know, isn't somethnig any big tech company does.

Again, the problem with your arguments here is that the standardized process at big tech companies is substantially better on those dimensions that you're complaining about than almost anything else out there. When you don't have a standard process and interviewers are just making random decisions based on no standard criteria and don't have to justify their decisions to anyone, people are absolutely going to get hired based on how they look.

Also:

This is called a system design interview and it's a standard part of every big tech interview loop (maybe except interns, not sure about new grads). This is also called whiteboarding. For sufficiently senior engineers, this is more important than coding but this is harder to administer and substantially less reliable. If you knew what was expected of you in this situation and you were given a challenging interview, I don't think you'd particularly enjoy this - it is hard and is far more arbitrary and less fair.

At a place like Google, Damore is the outlier by a fair margin. Top tech companies are quite liberal, especially white men. A lot of ethnic minorities are usually more apathetic/conservative, but still substantially more liberal than not. It's socially more acceptable to be a communist agitating for a revolution than a moderate republican.

This isn't how it works. First of all, you're being evaluated on a bunch of things, only one of which is coding. You can give an amazing solution and still fail the interview. No one cares about thinking outside the box, no one cares about your elegant visionary solution. It feels to me like you're sort of projecting based on how you would evaluate as an interviewer, but the reality is that the interviewer is capturing, almost mechanically at times, the meaningful aspects of your performance so that the totality of your performance can be evaluated by people who weren't there. You're not trying to impress the interviewer - you're trying to help them build a body of evidence such that hiring you can be justified to people who did not see you perform based on objective evidence.

At places that don't have a particularly rigorous hiring process, interviewers can make arbitrary decisions without having to justify but to me, that seems to be what you're arguing for.

I've literally heard hiring managers about some cool solution a dev came up with on the whiteboard. The Netflix came out and said time was a component. Of course time matters and thinking outside the box matters.

I didn't say they were the only things that matter. But when you're 10 minutes in and flailing some - you know you're not doing kicking ass - no matter how well you explain your process and show how you think. Some dev who aces the thing in 5 minutes and comes up with a really interesting solution is *of course* going to have a big edge.

That's so obvious I'm surprised I have to argue it.

I mean that's basically what an internship is. Some tech companies offer a version of that for experienced engineers too but I think a lot of that is mainly for underrepresented minorities and/or people in special circumstances (for example, mothers returning to work). But you can't do this for senior engineers - you can't trust unvetted people to lead big important mission-critical projects and if they are not delivering on big important mission-critical projects, you can't tell whether they are qualified to be senior engineers.

Lol "behavioral interview". Call it whatever you want - it's the same thing - a chance to weed out someone you don't want to hire. No company on earth is going to go "you know, I don't really like this guy - but he aced his whiteboards so we have to hire him, that's just objectivity".

And the reasons they don't like you could be very different from the reasons they give, and maybe even some bias they're not aware of.

Whatever happened to my coworker - I'm extremely suspicious it's either his age or his thick Mexican accent. The guy passed all the technical interviews and is one of the nicest guys I've worked with.

You're not giving any brand new dev the keys to a mission critical project w/o some vetting and oversight - no matter which way you hire them.

Plenty of companies hire senior engineers and architects on a contract-to-hire basis. It can easily be done.

If hiring managers are giving coding interviews, you likely don't have a rigorous interview process. Also, if your interviewers are often seeing "cool solutions" to some question, it's quite likely that at least one out of the question, the interview training or the interviewer is quite bad. Good interview questions are not supposed to have cool solutions that interviewers don't know about, interviewers are expected to understand just about every reasonable way a question can go and to steer candidates away from unexpected directions, should they arise (since they are almost always wrong).

You seem to be arguing against a standardized, rigorous interview process with evidence almost entirely gathered from interviews that weren't following a standardized interview process.

No one said time doesn't matter, of course it does for the obvious reason that the interviewer can only observe you for a finite amount of time. Thinking outside the box generally does not matter in a coding interview. If anything, it could hurt, especially if you're wrong, but the interviewer incorrectly believes that your outside the box thinking has merit and lets you waste time on it. Generally speaking, for most coding interviews, you will tend to be steered towards one of several possible known paths. Again, a well-conducted coding interview is not one where you're given a question and you just solve the question, impressing the interviewer with your coding/problem-solving prowess. You're communicating with the interviewer the entire time and how you handle the communication is probably more important than the code you write, assuming some basic level of competence and preparedness.

But what is it that you're actually arguing for that's better? You seem to be advocating that people just talk to you, not challenge you technically and make a random decision based on no real information, hopefully in your favor. How is that not purely a "do I like this guy or not" test?

For a real behavioral interview, interviewers are generally not making hiring decisions - they are for the most part documenting and summarizing what happened in the interview. I suppose a particularly malicious interviewer can lie about what happened but why? They are not going to work with this person either way and plausibly making up what happened in an interview is a lot of effort. Interviewers are simply not going to care one way or another about the candidate, certainly not enough for them to deliberately sabotage you. It's not easy to just keep up with the facts in the interview - one nice thing about a standardized process where the interviewer has to document what goes on and will often feel they are evaluated on how well they do this is that they don't really have as much energy left to focus on the less important stuff. It's a lot of work. There's definitely some projection here in that people who are used to interviewing much less rigorously are assuming that the level of capricious decision making that's present in their own interviews must always be present elsewhere.

This is wrong on two levels. First, the analogy I made had nothing to do with relevance - the point here is that even if the test wasn't relevant at all, playing ball is certainly evidence that you're motivated and conversely, not playing ball is certainly evidence that you simply don't care, which bodes poorly.

Second, it's not even close - a whiteboard interview is substantially more relevant to your work performance than your combine results. It's not even in the same ballpark. The gap grows further if you consider that teams have a huge amount of information on the players that's publicly available and well-scrutinized. They can probably draft almost as well without any of these silly workouts. Top companies are going to make an offer almost entirely based on your interview performance and they are almost always right when they do this - very few people who get the offer turn out to be unable to perform. Football teams don't draft you purely based on your workout results - for example, consider if you're a freak athlete with almost no football experience - it's always a combination of prior production guided by some additional information provided by workouts.

Even just qualitatively speaking, if you can code and design systems on a whiteboard, can talk intelligently about working well with others, you probably have most of the skills to do well at a top tech company. Combine results simply cannot demonstrate how you will do in real game situations - there are lots and lots of workout warriors who just don't have the football IQ to produce consistently.

For most football players, we're not talking about millions of dollars. The minimum wage in the NFL is 480K and for practice squad, it's like 130K. And a lot of these players have zero job security. The average career length in the NFL is 3.3 years and the median salary is 880K. Obviously stars make more (as do star engineers) but for lots of people at the combine, they aren't looking at something substantially better than ordinary senior engineers at top tech companies, who make like 400K-600K a year without risking their body, with substantially better job security and long-term career stability.

Then they are acting as junior engineers until you do. Which is why these types of programs (internship and others) usually yield junior and mid-level engineers, but not senior engineers. After all, why would performance as a junior engineer be enough to secure a full-time role as a senior engineer? Also, what problem are you solving here? Even if top tech companies were to do this, they still can't just hire anyone - there's no way they can accommodate everyone that wants a job - and they will have to use similar criteria to filter candidates anyway, with a slightly lowered bar. Also, are you sure that there's a large enough talent pool that merit this extra consideration? If you lower the bar slightly, you're just gunning for the talent that works at a similar tech company, just a tier below in terms of pay - would the best of them really jump ship for a contract-to-hire opportunity? Internships make sense in part because a huge number of people graduate from school every year, just about everyone will need a job and it's something students are expected to do.

But we've established that most companies are full of bullshitters - are you sure this isn't one of the possible channels for them? Also big tech companies are more successful than other companies on just about every metric - "other companies do it" is a very poor reason.

I think you're underestimating the degree to which these interviews are effective precisely because they can be prepared for. I used to think these are about skills and ability, but I think they work well in part because it effectively tests for a combination of natural ability, existing skills as well as the willingness to prepare for a technical interview. Ultimately, to be a high performer, you need some ability and skills but just as important is the willingness just to do whatever it is that needs to be done, especially in a technical context, to get things done.

I find that it's surprisingly hard for most people to consistently do what needs to be done - so many people have weird reasons for not doing what needs to be done, even things that aren't hard at all. And people's tendency in this regard doesn't even necessarily translate from one domain to another. And at least for engineers, the willingness to study for coding interviews, I think is a decent proxy for the willingness to do whatever it takes in a variety of technical contexts. At least among people that I know, that's what stands out about the people that are able to pass these interviews, as opposed people that aren't. They are not smarter, but they do whatever they need to do technically, without excuses.

Think of something like college admissions - I mean the entire process is a whole bunch of nonsense that has nothing to do with anything. Yet people who do go to top schools, on average, do far better in life, even if you control for all kinds of other stuff. People who don't even go to top schools, but had the ability to do so (based on test scores, grades, etc) do far better in life, again, controlled for other stuff. And we know your SAT scores can be improved dramatically with some studying, grades obviously even more so, and a lot of extra-curriculars are more about showing up than any kind of actual ability.

Yet all this nonsense is still quite predictive. I think this is due to these achievements being highly correlated with grit and effort. If you're the type of person who's willing to try hard in high school to go to a better school, you will also be trying hard later in life. Obviously not everyone's is going to stay the same - there are prodigies that fall apart later in life and late-bloomers that figure things out later in life, but on average, it's going to be quite predictive.

Yes there is a lot of BS people have to put with at places they work at.

If you can back this up quantitatively with

links to data AND/OR
sources with data we can read about it at

that would be greatly appreciated.

The red flag isn't really that you can prepare for them. It's if you *have* to prepare for them. There are a whole lot of people that make good/great engineers that would do poorly on these interviews without spending time preparing. Which means that they're not a great signal for the overall qualities you actually want to hire.


Sure, but there's a wide gap between doing what needs to be done at work and doing all of that in an existing job and then adding the time it takes to prepare for an interview. This extra effort will filter out a large set of people (a set that likely skews older, female, and poorer) that would be great hires but don't have that extra time. It's not that these types of questions are a poor filter - it's just that there's a clear red flag that it's not a particularly efficient or effective way of figuring out if someone is going to be a good hire. I don't have a better - scalable - way though.

I'm not American so definitely not familiar with the details of things like US college admissions / SATs. But I'd suspect there are a lot of the same issues here that I mentioned above. That is that not everybody has the time/effort/opportunity for doing this extra work (SAT prep, extracurriculars, etc.).

For example, take 2 people that spend the exact same amount of time on core school work, sleeping, eating, and leisure activities and have 30 extra hours/week. If one of those people needs to work 25 hours/week to help their family survive and the other gets to spend that doing bullshit extracurricular activities - the current process is going to find the working person is "trying less hard". But that's really not true.

So, yes, I think any process like this has a pretty giant red flag that would be good to solve.