Tales from the Interview: Monitoring the Situation

Joe T recently decided it was time to go job hunting. This mostly meant deflecting emails from head-hunters, doing phone interviews with ignorant HR departments, and the occasional on-site interview with a possible employer. One of those on-site interviews brought him to an IT services company which handled a few large US government contracts.

The building itself was buried way in the back of a tech park, behind a small pond inhabited by ducks and migrating geese. It was modern, in a circa 2002 way, gleaming white, with a large and mostly empty parking lot, and a glossy lobby and a pair of cheerful receptionists.

One of the receptionists made a call. A few moments later, Ricky, Joe's contact, stumbled out of the elevator. He had the look of a man who hadn't slept in a month. As they piled into the elevator, he said, "We are really eager to fill this position. Just between you and me, I think you've got it sewn up, but you need to meet the team."

Ricky led him into a conference room, where a few other sleep-deprived looking techs nursed their coffees. After a round of handshakes, they dove into a series of sysadmin/operations questions. Some of them were just dumb trivia: "What's the purpose of the -i flag on the ls command?" Some were dumb soft-balls: "What time management techniques do you use?"

After the warmup, Ricky leaned forward, and got right to the meat of it. "You've got three data centers," he said, "how would you monitor all three of them?"

That was a vague question, so Joe could only give a vague reply. "Well, I'd imagine all three are connected to the same management network. So I'd run some sort of monitoring package on that network."

"You don't have that," Ricky said. Joe couldn't be sure, but he thought he might have seen a grimace on the face of one of the techs.

"Well," Joe said, "then we'd just have to set up monitoring in each DC independently, and then aggregate somehow, but that's really not the best option. Setting up a management network is-"

"Ah," Ricky interrupted, "but what if you don't have the budget for any monitoring software?"

Joe, sitting in the middle of the interview, jumped to a conclusion about Ricky's line of questioning. This was one of those old interview tactics: no matter how the candidate replies, throw up an obstacle, and see how they can "think outside the box". This was just a bizarre hypothetical situation. He played along. "I guess we'd need to install Nagios or some other open source monitoring tool."

"The customer doesn't allow any open source software on the network," Ricky countered.

"None?" Joe asked. "But don't you have some Linux servers?"

Ricky nodded. "We do. But no open source software on the network."

"Well, I mean, if you can't buy software, and you can't use free software, you can't really do any meaningful monitoring."

Now Joe was certain: the techs were all grimacing, and oh so very slightly nodding along with him. This was what they expected him to say, but Ricky wasn't going to give in yet. "Now, you can't just throw your hands up and say, 'It can't be done.' You have to monitor these datacenters, so what do you do?"

"Well" I mean, at that point, we're just writing our own scripts and running cron jobs. But that's a totally ridiculous scenario. No one would actually run a datacenter that way!" Joe punctuated the statement with a laugh, because obviously this was all just silly.

No one else laughed. Ricky's line of questioning wasn't a hypothetical, it was an accurate description of their infrastructure in their datacenters. After a solid minute of awkward silence, Ricky started asking questions about the previous jobs on Joe's resume.

Somehow, Joe left the interview with a job offer, but he politely declined.

[Advertisement] Manage IT infrastructure as code across all environments with Puppet. Puppet Enterprise now offers more control and insight, with role-based access control, activity logging and all-new Puppet Apps. Start your free trial today!