The Defensive Contract

Working for a contractor within the defense industry can be an interesting experience. Sometimes you find yourself trying to debug an application from a stack trace which was handwritten and faxed out of a secured facility with all the relevant information redacted by overzealous security contractors who believe that you need a Secret clearance just to know that it was a System.NullReferenceException. After weeks of frustration when you are unable to solve anything from a sheet of thick black Sharpie stripes, they may bring you there for on-site debugging.

Beforehand, they will lock up your cell phone, cut out the WiFi antennas from your development laptop, and background check you so thoroughly that they'll demand explanations for the sins of your great-great-great-great grandfather's neighbor's cousin's second wife's stillborn son before letting you in the door. Once inside, they will set up temporary curtains around your environment to block off any Secret-rated workstation screens to keep you from peeking and accidentally learning what the Top Secret thread pitch is for the lug nuts of the latest black-project recon jet. Then they will set up an array of very annoying red flashing lights and constant alarm whistles to declare to all the regular staff that they need to watch their mouths because an uncleared individual is present.

Then you'll spend several long days trying to fix code. But you'll have no Internet connection, no documentation other than whatever three-ring binders full of possibly-relevant StackOverflow questions you had forseen to prepare, and the critical component which reliably triggers the fault has been unplugged because it occasionally sends Secret-rated UDP packets, and, alas, you're still uncleared.

When you finish the work, if you're lucky they'll even let you keep your laptop. Minus the hard drive, of course. That gets pulled, secure-erased ten times over, and used for target practice at the local Marine battalion's next SpendEx.

Despite all the inherent difficulties though, defense work can be very financially-rewarding. If you play your cards right, your company may find itself milking a 30-year-long fighter jet development project for all it's worth with no questions asked. That's good for salaries, good for employee morale, and very good for job security.

That's not what happened to Nikko, of course. No. His company didn't play its cards right at all. In fact, they didn't even have cards. They were the player who walked up to the poker table after the River and went all-in despite not even being dealed into the game. "Hey," the company's leaders said to themselves, "Yeah we'll lose some money, but at least we get to play with the big boys. That's worth a lot, and someday we'll be the lead contractor for the software on the next big Fire Control Radar!"

So Nikko found himself working on a project his company was the subcontractor (a.k.a. the lowest bidder) for. But in their excited rush to take on the work, nobody read the contract and signed it as-is. The customer's requirements for this component were vague, contradictory, at times absurd, and of course the contract offered no protection for Nikko's company.

In fact, months later when Nikko-not yet aware of the mess he was in-met with engineers from the lead contractor-whom we'll call Acme-for guidance on the project, one of them plainly told him in an informal context "Yeah, it's a terrible component. We just wanted to get out from under it. It's a shame you guys bid on it""

The project began, using a small team of a project manager, Nikko as the experienced lead, and two junior engineers. Acme did not make things easy on them. They were expected to write all code at Acme's facilities, on a network with no Internet access. They were asked to bring their own laptops in to develop on, but the information security guys refused and instead offered them one 15-year-old Pentium 4 that the three engineers were expected to share. Of course, using such an ancient system meant that a clean compile took 20 minutes, and the hidden background process that the security guys used to audit file access constantly brought disk I/O to a halt.

But development started anyway, depsite all the red flags. They were required to use an API from another subcontractor. However, that subcontractor would only give them obfuscated JAR files with no documentation. Fortunately it was a fairly simple API and the team had some success decompiling it and figuring out how it works.

But their next hurdle was even worse. All the JAR did was communicate with a REST interface from a server. But due to the way the Acme security guys had things set up, there was no test server on the development network. It wasn't allowed. Period.

The actual server lived in an integration lab located several miles away, but coding was not allowed there. Access to it was tightly-controlled and scheduled. Nikko found himself writing code, checking it in, and scheduling a time slot at the lab (which often took days) to try out his changes.

The integration lab was secured. He could not bring anything in and Acme information security specialists had to sign off on strict paperwork every time he wanted to transfer the latest build there. Debuggers were forbidden due to the fears of giving an uncleared individual access to the system's memory, and Nikko had to hand-copy any error logs using pen and paper to bring any error messages out of the facility and back to the development lab.

Three months into the project, Nikko was alone. The project manager threw some kind of temper tantrum and either quit or was fired. One of the junior engineers gave birth and quit the company during maternity leave. And the other junior engineer accepted an offer from another company and also left.

Nikko, feeling burned out and unable to sleep one night, then remembered his father's story of punchcard programming in computing's early days. Back then, your program was a stack of punchcards, with each card performing a single machine instruction. You had to schedule a 15-minute timeslot with the computer to run through your program which was actually a stack of punchcards. And sometimes the operator accidentally dropped your box of punchcards on the way to the machine but made no effort to ensure they were executed in the correct order, ruining the job.

The day after that revelation, Nikko met with his bosses. He was upset, and flatly told them that the project could not succeed, they were following 1970's punchcard programming methodologies in the year 2016, and that he would have no part in it anymore.

He then took on a job at a different defense contractor. And then found himself working again as a subcontractor on an Acme component. He decided to stick with it for a while since his new company actually read contracts before signing, so maybe it would be better this time? Still, in the back of his mind he started to wonder if he had died and Acme was his purgatory.

[Advertisement] Manage IT infrastructure as code across all environments with Puppet. Puppet Enterprise now offers more control and insight, with role-based access control, activity logging and all-new Puppet Apps. Start your free trial today!