Tested Backups

The "Big Boss" of Initech's Australian division ran the Sydney office as his own personal kingdom. Work- or workers- he didn't care for was banished to the hinterlands of the Melbourne office. For example, IT services was a "useless sack of morons who only know how to spend money," and thus the entire department was banished to Melbourne.

Stewart C. lived in Melbourne, and was a new hire not long after the exile. The Melbourne office, with a 900km buffer zone protecting it from the whims of upper management, was actually a decent place to work. At least, until Brendan arrived.

Brendan was an upwardly mobile middle manager based out of Sydney. One day, he did something to offend Tony. The scuttlebutt had a lot of rumors about what that was, ranging from "oh, he was late to a big meeting," to "he took the last beer at the company party". Rumors aside, the facts were clear: Brendan was in Melbourne with a long list of projects to manage, and he wasn't going back to Sydney until he completed all of them.

A few days into his exile, Brendan brought that list of tasks to Stewart's cube. "Stewart. Stew. Stewie. I'm gonna need you to go ahead and test our backups, yeah? Make sure we can actually restore from them, right?"

"That's a great idea," Stewart said. "So great, that we test them on a regular basis. I can send you a doc that covers our backup strategy-"

"Right, yeah, but have you tested them in production?" Brendan asked.

Stewart paused, making certain that he'd heard the question properly. "Well, no""

"Right then, yeah? They haven't really been tested 'till we run them against production, yeah?"

"We" really shouldn't test things in production," Stewart said. "I mean, what if the test fails?"

Brendan shook his head and laughed. "Stewie, why would the test fail unless the backups are broken, right? Yeah, I'm gonna need a plan to cycle through every production server. Wipe it, restore from backup tape, and confirm it's working, yeah? Right, so get on that. Orders from Tony, yeah, we need this by next week."

Stewart immediately got to work- on making sure this hare-brained idea didn't go forward. He roped in his boss, who roped in her boss, and everyone was on the same page: this was a terrible idea that had no real benefit and carried with it too much risk and downtime. "Buuuuuuut," his boss said, "it's what Tony wants. We haven't got the political clout to tell him 'no', so we're going to have to do it."

Wind of Stewart's attempts at killing the project got back to Brendan. This resulted in a rather nasty meeting invite demanding a "status update". Stewart wasn't the only invitee. When he took a seat next to Brendan, they were both staring at a speakerphone.

The phone resonated with Tony's booming voice and confident authority. "Well? What's all this then?" he demanded.

"I'll tell you what this is," Brendan said. "It's just like you said, sir- everyone out here in Melbourne is lazy and obstructive, right? Yeah."

"You're telling me crap I already know," Tony said, his voice crackling on a bad connection. "Of course they're useless, that's why they're out in Melbourne. Was there a point to this call?"

"Well, yeah, right," Brendan said. "I wanted you on the line for this because Stewart has been the worst, right? He's been nothing but a trouble-maker disrupting my efforts and second-guessing me-"

Stewart started to mount a defense, but he couldn't get a word in before Tony's laughter drowned him out. "Brendan, if you can't handle the Melbourne office, how do you expect to cut it back here in Sydney? You just haven't got the guts for this business, Brendan. Now, c'mon then, you've got your orders. Chop chop, and no excuses."

Tony ended the call, leaving Brendan and Stewart staring at each other in uncomfortable silence. Brendan was the first to break the silence. "Yeah, as you can see," Brendan said, "this project has attention right from the top. So you're going to give me a workable plan for testing all of the backups on the production servers by tomorrow- or else, right? Yeah."

Stewart nearly demanded, "Or else what?", but decided against it. Instead, he wrote up his plan, noted the steps that involved downtime, and sent it to Brendan the following day. Brendan didn't have any follow up questions, and nobody told Stewart to execute the plan, so he promptly forgot about it and got back to doing real work.

Until all IT operations came screeching to a halt two weeks later. Brendan, as it turns out, decided he didn't trust Stewart to run the tests. Instead, he strong-armed one of the junior technicians into doing it for him- but not until after making his own modifications to Stewart's plan. Specifically, he wasn't happy with the schedule- Stewart had very conservatively scheduled only one server to be tested a week, outside of regular business hours, so that any failures or issues would be minimized. That would have taken months, and Brendan wasn't willing to wait that long, so he made the junior technician do them all at once. Without coordinating downtime.

The result was every bit as catastrophic as one might expect, especially considering the junior technician didn't think to order the backup tapes before starting the test, which meant they needed to be shipped from their offsite location. All in all, they lost two days of operations and a lot of reputation among their customers. Tony was furious, and he knew exactly who to blame: the "useless sack of morons who only know how to spend money."

Obviously, the folks in Melbourne couldn't handle the complicated job of running IT services, so the IT department there was closed down at the end of the quarter and Initech hired on contractors to set up a new datacenter in Sydney. Stewart wasn't particularly happy to lose his job, but he took some solace in the fact that he had prospects lined up- while Brendan didn't.

[Advertisement] Manage IT infrastructure as code across all environments with Puppet. Puppet Enterprise now offers more control and insight, with role-based access control, activity logging and all-new Puppet Apps. Start your free trial today!