Deep Fried Offshore

Stephen worked for an Initech that sold specialized hardware: high-performance, high-throughput systems for complex data processing tasks in the enterprise world, sold at exorbitant enterprise prices. Once deployed, these systems were configured via a management app that exposed an HTTP interface, just like any consumer-grade router or Wi-Fi access point that is configurable through a website (e.g. 192.168.0.1).

Stephen worked with a diverse team of American engineers who were finishing up the management application for a new model. The product was basically done but needed a little bit of testing and polish before the official release. They expected several months of post-release work and then the project would go into maintenance mode.

Then disaster struck. A pointy-haired boss somewhere up in a fuzzy area of the organization chart simply labeled "Here be VPs" discovered the large salary difference between American engineers and off-shore workers, and decided American engineers were far too expensive for software "maintenance". The company decided to lay off 300 software engineers and hire 300 fresh-out-of-college replacements in a foreign country with much lower labor rates. The announcement was overshadowed by the fanfare of the product's release and proudly billed as a major "win" for the company.

Stephen was lucky enough to stay on and shift to other projects, the first of which was to assist with the transition by documenting everything he could for the new team. Which he did, in hundreds of pages of explicit detail, explaining how to use the source control repository for the project, execute and log unit tests, and who to contact when they had questions regarding the hardware itself. After that, he devoted himself to other projects. Months turned into years and Stephen assumed by the silence that the handover was successful and he'd never see the management app again.

Of course that isn't what happened.

There's an old joke on the Internet called "How to shoot yourself in the foot" (http://www.toodarkpark.org/computers/humor/shoot-self-in-foot.html) that lampoons the complicated process of shooting yourself in the foot in various programming languages. Here is one such entry:

370 JCL: You send your foot down to MIS and include a 300-page document explaining exactly how you want it to be shot. Three years later, your foot returns deep-fried.

Three years after the management app was passed off to the new offshore engineer team, a new panic was boiling through Initech. The management application was as slow as molasses, buggier than flies in soup, with a user interface that was battered and deep-fried in a nonsense language that vaguely resembled English. It also crashed a lot, each time requiring a power reset to bring the system back up.

Initech had simmered along by only shipping the original version, but now they had this expensive, powerful product with three years worth of hardware and firmware updates that the management app could not configure and thus were not available to customers. Several important customers canceled their support contracts rather than pay for half-baked updates that rarely worked, and many prospective customers passed them right by when basic features printed in the product brochure were "unavailable for demo". They were falling behind in the industry.

It was bad enough that management finally decided to do something.

That something was to toss the offshore team and see how many of their old laid-off engineers they could scoop up, which was, not surprisingly, few. As one of the few original engineers who still worked there, Stephen was whisked away from his current projects to assist.

With a bite-sized portion of the old team re-assembled, they set to work to unscramble the situation. Stephen noticed that the last source control check-in was from three years ago. Upon further inspection, he realized that not one of the offshore engineers had ever committed.

He called up the offshore office to see if they had their own repository. The last few remaining offshore workers didn't know, and when told to search every system they had, they replied they couldn't because their local manager had sold all of their computer equipment on an auction site upon learning of their impending layoffs.

He then contacted the hardware team in hopes that maybe somebody had a clone of the offshore team's repository, but the hardware guys claimed the software team was laid off three years ago and they were not aware of an offshore replacement team.

With no current source code, and an old repository that hadn't been committed to in three years, Stephen and the team were left with no choice but to pick up exactly where they left off three years ago"

[Advertisement] Manage IT infrastructure as code across all environments with Puppet. Puppet Enterprise now offers more control and insight, with role-based access control, activity logging and all-new Puppet Apps. Start your free trial today!