The Backup Pipeline

Dick was the main man in charge of his homegrown facilities-management system known as Q-Max. His brainchild was utilized across a large office complex to enter and track building maintenance tasks. Whenever a sink was broken, toilet clogged, or a foul-smelling science experiment had to be exterminated from a fridge, Q-Max was there to track and route the incident.

While that was its original intent, Dick wound up selling the office managers on expanding its role to include reporting that would help them make budgetary decisions. It helped them determine which maintenance guys deserved a raise and which buildings needed renovations the most. So despite the maintenance department despising Q-Max because it made them do stuff, management loved every bit of it. Little did they know how precarious the system was on the backend.

Dick focused on keeping this amazing application running, and let the company's "Disaster Recovery Team"- one guy named Pete- worry about the worst case scenarios. They kept a regular offsite backup of its data so that in case of something catastrophic like a meteor strike, Q-Max would not go the way of the dinosaurs. Their backup system was quite simple, yet had a necessary workaround step - Every night around midnight an Oracle backup was created on the database server. From there, they copied to an offsite tape backup. Due to infrastructure limitations- bandwidth and reliability, mostly- this mirroring process had to run to get the backup to a location the tape robot could get its iron clutches on.

Eventually Dick and Q-Max grew even bigger and got a larger budget to spend on infrastructure. Some network upgrades allowed Pete to have a backup system which could copy a backup directly to the tape library, thus eliminating the need for that silly mirroring job. The new system was implemented and taking nightly backups was a breeze.

A few years passed as Q-Max continued to have its automated backups taken every single night. One cold winter's eve, an old pipe that happened to run over the office's server room froze and burst. One of the many casualties of the unplanned cold shower was the database server that housed Q-Max. Pete called Dick frantically one night while he was home sipping hot cocoa.

"DICK, I NEED YOUR HELP! There's water everywhere!" Pete shouted to him. "It's raining in the server room!"

Thinking Pete was delusional, Dick calmed him down to get the full story. The loss of the database server and several others was troubling, but at least they still had their trusty offsite backup tapes! "Man, that sucks. I'll come in to help clean up and salvage what we can. In the meantime, put in an urgent request to have the last backup tape delivered first thing in the morning!"

Since Q-Max was down and there were no maintenance people to be found, Dick and Pete spent the night with mops and buckets cleaning up the pipe's mess. Around 7 AM, Dick was paged down to the lobby for a delivery. "The tape is here!" he exclaimed with joy, eager to get Q-Max back online.

Dick brought the tape back to the now-dry server room and prepared to restore from it on an alternate database server. The tape directory stared back at him blank. "That's odd, there's nothing in here. They must have sent the wrong tape!" Dick put in an angry call to the company that housed their tapes. They checked other tapes that had been used for Q-Max backups only to find all of them were blank.

Furious, Dick could think about nothing but lawsuits against the company that screwed up their entire backup system. In the meantime, he dispatched Pete to prove the backup job on their side was working. Pete ducked out for an hour or so. When he returned, he looked like he'd just been diagnosed with some horrible disease.

"I checked out the backup job and, well, you're not going to like this..." Pete muttered nervously.

"Ok... what exactly happened?" Dick asked, unsure if he actually wanted the answer.

"Well, the good news is our backup job is working fine. However... Do you remember long ago when we had that stupid mirror job to copy the local backup to the external tape drive? I kinda, sorta, forgot to disable that. So after our new backup job runs, the old one copies an empty directory over top of it, and the legitimate backup is lost. It's been like that. For years."

If there was any remaining water from the pipe bursting, it would have turned to steam as Dick became red hot with anger towards Pete. Dick would have to strangle him later, though. First he had to figure out how to rebuild Q-Max's data from the ground up.

[Advertisement] BuildMaster is more than just an automation tool: it brings together the people, process, and practices that allow teams to deliver software rapidly, reliably, and responsibly. And it's incredibly easy to get started; download now and use the built-in tutorials and wizards to get your builds and/or deploys automated!