The White Appliphant

Circa 2010, Becca's employer, Initech, was growing. "Growing", in this case, meant "acquiring competitors who had niche products that their customers wanted added to Initech's portfolio".

One such product was a content-management/workflow tool, already sold to some big-name, multi-billion dollar companies. The tool fell into that niche between "really useful to the people who need it" and "buggy as an ant hill inside of a termite mound under a wasp nest". Sales were good, but the devs were underwater and the backlog of feature requests and bug fixes were growing. So Initech bought the vendor, fired most of the developers, and handed it to an Initech team.

"It's just some bugfixes," management said, "what could go wrong?"

The first thing that went wrong was the senior dev assigned to the project rage quit after two weeks with it. "I ran the install scripts, which are supposed to provision a new database and deploy the web app files to a web server. There are thousands of interlocking scripts to make this happen, and half of them don't work. Along the way, it creates hundreds of SQL tables, all with names like table1, or important_table1, and many of them are never queried by anything in the code, despite having data." What followed was an ultimatum: take me out of this project or I'll take myself out of the company.

Given that this senior dev was central to many other projects, the company shuffled things around and put another sacrificial lamb in place, a different senior dev. This dev needed to add a small feature- a new field to a screen. Six weeks later, the dev had something that mostly worked. It was a hacked on retrofit- there was no "right way" to add a new field to a screen; any modifications to any of the front end or back end or database code tended to cause explosions, thunderstorms, and the gnashing of teeth. This dev, also very high up and respected, repeated the ultimatum.

When adding a new field to a screen takes six weeks and burns a developer out, you can imagine how hard it is to fix bugs. The junior devs put on bug fixes were burning out just as quickly, but producing far fewer results. The bug backlog kept growing far faster than bugs could get patched.

if(hasrating){thechkbox = currRating[0].parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.children(2).children(0).children(0)thechkbox.disabled = false;}

That was some of the better code in the application. Much of the code was a bizarre mix of Classic ASP server-side code and JavaScript- with the ASP code frequently generating the JavaScript. It also generated SQL queries through string concatenation. It'd often stuff those queries into data- attributes in HTML elements because that was the only way it new how to link data together in the UI. Many times, raw SQL queries would get placed as GET parameters on the URL.

As you could imagine, this meant plenty of options for SQL injection attacks, but they were rarely needed: your privilege level was stored as text in an <input type="hidden">. Anyone who knew how to use the browser debugger could promote themselves to Admin.

Staring at this, the company decided to do the natural thing: dedicate all of their developers to a gigantic, ground-up, rewrite. "If we put everybody on this, it'll be done in, what, three months? We can totally do this."

Well, they couldn't put everyone on it full time- the existing package was so buggy and dangerous that developers constantly needed to be pulled back into the old version to just keep it running. But also, when trying to understand the requirements for a rebuild, they rapidly discovered that every customer had a different version, with different features, and completely different needs.

The original developers had never used source control. When they built the software, they built it for one customer, and deployed it. When the next customer asked for the software, they copied all the files to a new directory, and tweaked and modified it to suit that customer. When the third customer came on, they looked at the versions they deployed for Customer A and Customer B, decided which one made a better starting point, and then copied-and-pasted for Customer C. That, by the way, was why there were hundreds of database tables that seemed to have no purpose: they didn't for this version of the software, but did for other customers.

The attempt at a ground up rewrite lasted all of two days before management admitted defeat. That was still too little, too late. The general awfulness of this new product, the fact that developers kept getting shuffled onto it and burning out in weeks, and the sense that the company had no plan to get to a reasonable software development lifecycle for it meant that anyone with experience jumped ship- including Becca.

Within three months of purchasing this white elephant, Initech had lost about 70% of its developers. Their core products started to be negatively impacted, and the company very nearly went under.

Now, so-called "vulture capital" investment tends to take loans to buy companies that aren't very profitable (but have plenty of assets). They transfer the debt to the company (making them liable for their own purchase), strip them of any assets of value, and then send them off to collapse under the weight of debt obligations they can't possibly fulfill. Venture capitalists get rich, companies go out of business.

Initech's "fix" for all of this was to steal a page from that playbook. They went into debt to buy another company with some promising products, transferred those products to Initech's portfolio, and took all of the tech-debt products and shoved them into the victim's portfolio. Initech kept a generous licensing agreement to continue to sell and host the broken product, but got to blame the "vendor" for its crappiness- and paid nothing to its ongoing maintenance or support.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!