Interviewing candidates

A quick note: I'm going to be traveling for much of the rest of June and I haven't got articles queued up, so the blog will go dark for a bit; see you in July!

In the last two episodes I did reruns of earlier articles on the technical interview process. I thought I might go into a little more detail about how I structure interviews and what I'm looking to get out of them.

I have some primary goals:

• Prevent bad hires
• Make good hires
• Leave the candidate with a positive impression of the company

Of course preventing bad hires is of far higher priority than getting good hires. If we fail to get a good hire, that's too bad, but if we make a bad hire, that can drag down the productivity of a team for years.

That last point is also key; if we want to hire the candidate then obviously they need to have a positive impression. But I want all the no-hire candidates to have a good impression as well. They have friends who might want to interview. They may be in a position to make purchasing decisions or product recommendations now or someday. An interview is a very expensive "high touch" business process; if we don't get a hire out of it, at least maybe we can get a customer, or if not that, at least some good will.

The actual interview goes like this.

First I'll make sure that the candidate is comfortable, has a drink, doesn't need to use the restroom, and so on.

Next, if I am not the first interview of the day then I ask "how's it going so far?"

The function of this question is to put the candidate at ease, get the conversation going, and see if they have any concerns about the process so far. In some cases I have information about how the candidate is doing, but it is not intended to be a "gotcha" question. (At Microsoft we were encouraged to give feedback ASAP to the other interviewers down the line so that they could dig into potential problem areas; at Coverity we are more inclined to have every interviewer start unbiased. Both approaches have their pros and cons.)

Then I pick something that looks vaguely interesting and fairly recent on the candidate's resume, and ask them to give me first the "elevator pitch" - a few sentences describing the project and why it was important or interesting - and then to describe the problem they had to solve in more detail.

This is, again, to put the candidate at ease because they are talking about something they chose to put on their resume and are presumably an expert in. I want to find out first, can they communicate well? You'd be surprised at the number of people who put things on their resume that they cannot describe the purpose of in a sentence or two. Second, what actually did they do on this project? This may come as a shock to you, but people inflate their resumes all the time. By digging into the details of what exactly they did on the project, I get a sense of whether they really are "smart and gets stuff done" or not.

Then we come to the really interesting part: the technical question. The purpose here is to figure out whether or not the candidate will be successful writing the kind of code that they're going to have to write every day. Now, I know all the criticisms of whiteboard coding questions, and I agree with them. It's an artificial environment, it's not how people actually code, it is high stress, the problems are unrealistic, and so on. I try to carefully design coding questions that minimize these difficulties while still telling me whether or not the candidate can actually do the job. (And if a candidate would prefer to not write code on a whiteboard, I can set up a computer as well. But the questions I ask seldom require writing more than, say, six lines of code; most successful candidates should be able to write a half-dozen short lines of code without help from their editor of choice.)

And it really is important. The jobs that I interview candidates for have an unusual set of requirements. Candidates for language analysis tools teams have to be strong in both theoretical computer science fundamentals and able to work on a real-world, imperfect codebases. I've had plenty of interviews with academics who understood theory deeply but could not write practical code, and with experienced industry coders who could not tell me what a binary tree was.

A good technical question has characteristics such as:

• can be stated and solved in a short amount of time with a short amount of code
• does not require an "aha!" insight that is difficult to come up with under stress
• is not a "stock" problem or "brain teaser"
• can be made arbitrarily harder or easier, to better gauge what level the candidate is really at
• is similar to the sorts of problems the candidate will have to solve on the job
• tests skills other than straight up coding, like ability to read existing code, or deal with an ambiguous specification

A problem that I used for a long time at Microsoft and works even better at Coverity has this basic outline:

First, here are two public API methods written in C++, each consisting of three lines of very straightforward code. The APIs handle "jobs" on a "server" and are called by a "client". The scenario is deliberately very vague, but the code is clear. I talk the candidate through each line and make sure they understand it. I'm looking for just a basic understanding of straightforward syntax here. (The vast majority of candidates list familiarity with C++ on their resume as that is a job requirement.)

Next, I tell the candidate that they are tasked with code reviewing this code for any problems whatsoever. I see what sorts of general problem areas they look for when reviewing code they did not write. The code I give has design, robustness, security, testability and portability problems. I see which of those areas they dig into naturally, and if they miss some, I suggest "what if the client is deliberately attempting to damage the server; what can they do?" or "what if I recompiled this code to run on a 64 bit machine?" and see if they take the hint.

This is a real-world skill that I have to use every day - first, in reviewing my own code and that of my coworkers. And second, because I'm in the business of writing programs that look at buggy code and figure out that it's buggy. If the candidate can't review other people's code then they're unlikely to be successful anywhere. If they cannot find any of a dozen bugs in a program that has only six statements, they're unlikely to be successful writing analyzers that find bugs.

Here I'm looking for comprehension, critical thinking and basic domain knowledge. Red flags often surface at this point. I've had candidates with PhDs in computer science, for instance, who did not know that on a 64 bit architecture, pointers are 64 bits wide. And why should they? As is often pointed out, astronomy is not the science of building telescopes. Knowing a lot about theoretical computer science does not necessarily entail knowing how wide the bus is between the processor and main memory. But a candidate who does not understand how pointers are implemented on typical machines is unlikely to be successful working on a team that designs analyzers that look for errors involving pointer size.

I also see a lot of candidates who know that code is wrong but cannot say what the consequences of it are. Undefined behavior is by definition undefined, but describing what might happen if a hostile client, say, forges a pointer to arbitrary memory and causes the server to execute a virtual method with that pointer as "this" demonstrates that the candidate understands how the compiler actually generates the code.

That's the first phase of the technical question. The next phase involves design, and maybe some small amount of coding. "You can change the implementation of these APIs but not change the method signatures; how would you fix each of the bugs you found?"

Again, this is a basic job skill that we have to exercise every day: fixing a bug in a callee without being able to change the behavior of the caller. Perhaps the calling code is owned by the customer, or another team, or has already shipped, or whatever.

At this point I am looking for things like:

• Are there any more red flags? Good heavens, the number of candidates I've seen who tried to fix the portability problem by compressing a 64 bit pointer into a 32 bit integer is enormous. There's no clever trick here that I'm asking them to figure out. You can't fit twenty pounds of flour into a ten pound sack, that should be obvious, so you need to find another way to solve the problem.
• What tools are in the candidate's toolbox? All candidates eventually figure out that they will need to add an auxiliary data structure that maintains a map from a 32 bit integer to a 64 bit pointer, because of the aforementioned ten pound sack. Do they know that there are off-the-shelf map classes? If not, do they have confidence that they could write one?
• Does the candidate identify the actually hard part of the problem? The hard part of the problem is not realizing that you need a map, but rather ensuring that the keys are unique when a new key-value pair is added to the map. (As I alluded to in the previous episode.) Even when I repeatedly ask "in your sketch of the new code you never say how you compute the value of the key; how is that value generated?" some candidates just don't follow up on it and keep on trying to tell me how the map works. Or they go back to the previous point and start trying to compress the 64 bit value into a unique 32 bit key, which is impossible, and is the reason why we have a map.
• How does the candidate deal with an ambiguous situation? The problem is deliberately vague, and the hard part involves generating unique values. If there are two clients each generating two jobs and then shutting down, generating unique 32 bit integer keys is not hard. If there are millions of clients generating millions of jobs then generating unique keys may be very difficult indeed. Many candidate never ask me how many clients and jobs there are in this system, and therefore some generate highly over-engineered solutions, and some generate solutions that do not scale at all.
• Can the candidate make the connection to an existing solved problem? This is hard to do under stress, but it is still interesting to see. Some candidates very quickly figure out that the problem "generate a unique 32 bit number and don't re-use it until the client is done with it" must have been previously solved by whoever wrote the 32 bit version of their favorite memory allocator. After all, that's what a memory manager is: a device which gives you a unique pointer when you ask for one, and does not give it out again until after you say you're done with it. Therefore the problem I'm posing can be reduced to a previously solved problem. Some candidates exhibit magical thinking about memory allocators and the like, as I alluded to previously. I have had many candidates realize that the problem is analogous to a heap; I've only had one candidate say "I'll solve the problem by having Windows make me a four billion byte heap, and then allocate single bytes out of it to generate unique 32 bit numbers". That is really reducing the problem to a previously-solved problem!

If the candidate is doing very well then I can easily make the problem harder. Suppose the clients are calling on many threads; what locking strategy keeps throughput high? Suppose the jobs need to survive the computer being rebooted. Suppose the server is on a cluster and can offload work onto other servers. I want to find out just how deep their knowledge goes. Or I can make it easier; suppose there is never more than ten clients. Suppose that jobs starting and finishing are in some reasonable order, like, a job started later than another always finishes later as well. And so on. I want them to walk away feeling like they solved at least some version of the problem.

Finally, I try to save some time at the end for the candidate to ask me questions about the team, the position, the company, and so on. (No candidate has ever said "could you write some code to reverse this linked list for me?" and I would be highly amused if they did. What is sauce for the goose is good for the gander, I suppose.) This is another chance for me to "sell" the company to the candidate, and also I get some insight into what is important to them from their questions.