A Lumpy Christmas
Every "enterprise" shop has that one system you hope you never need to touch. It's older than you are, "documentation" consists of whispers and rumors about its behavior, and it is absolutely 100% business critical. If it goes down, the business goes down.
Fortunately, you'll never have to touch that system, because there's an Ancient Wizard who has been sitting in the same cube since 1973, and knows its secrets. As long as the Wizard is around, you'll never touch it. Of course, if the system goes down when the Wizard is out of the office" well, fixing that would require a Christmas miracle.
Renee was a programmer-elf in Santa's Workshop. It was a huge organization with many layers of middle-elfagent, so 90% of her job was sitting in pointless meetings with no agenda. In the remaining 10%, she occasionally added features to their in-house legal invoice processing software. Since nearly every present they shipped was an elf-produced knockoff of a large company's trademarks, legal billing was one of the Workshop's largest line items.
There were other important IT systems. Supply chain was massive. All raw materials, aside from the naturally-occurring candy canes, had to be imported. Whether it was fabric for producing dolls, or ABS plastic and injection molds for producing dolls action figures. While shipping had a special organizational importance, IT rarely was engaged. The CEC (Chief Executive Claus) had learned to do pivot tables and VLOOKUPs in Excel, and did all his shipment planning himself.
The 100% business-critical mystery system was NNOSE: Naughty/Nice Observation and System Evaluator. It was the "source of truth" for the all important TPS reports: Total Presents Sacked. The Ancient Wizard who cared for it was literally an Ancient Wizard, Merlin. And yes, Merlin really was Santa's sidekick. NNOSE ran on a piece of NORAD surplus computer hardware and was programmed in a dialect of MUMPS nicknamed "Sugar LUMPS" or just "LUMPS". Merlin was the only one who knew how the system worked and what the system did, and Merlin was the only one who was ever supposed to touch it.
As the clock clicked closer to 12/24, the shipping deadline, the Workshop got busier and busier, while software developers got less and less busy. Since time was critical, no software changes went into production unless there was an absolute show-stopping bug. The software teams basically were on an extended break until the New Year, and no code releases would happen until mid-January at the earliest.
So, during the busiest, most critical time of the year, the development team mostly spent their vacation hours. Merlin was out in Avalon, trying to patch things up with his ex. The senior Project Elfagers were on a retreat in Cancun. All through the shop, not a developer was stirring, aside from Renee. Renee loved to work during the quietest time of the year, when no one scheduled any meetings. It was its own kind of vacation"
" at least until Hermie, the lead developer who really wanted to be an MBA, popped his head into her cube.
"NNOSE is down. The Big Guy just tried to pull the final TPS report for the year, and not only did it not have a cover sheet, it printed ten pages of errors instead of data. I've already tried to raise Merlin, but he's not on Slack, and there's no cell service in Avalon."
If the TPS report were wrong or unavailable, Santa couldn't accurately match kids up with presents. Naughty kids could get great gifts, nice ones might get useless garbage like an Amazon Echo or Google Home. He needed that report.
"Okay, but I know the report has been run previously," Renee said. "Can't they just run with an old version?"
"There aren't any old versions. TPSes get shredded after review, ever since GDPR went into effect."
Against her better judgment, Renee set down her hot cocoa and said, "I'll see what I can do."
Merlin didn't keep any digital documentation for NNOSE. Everything about the system was stored in binders full of parchment in his cube. Calling the contents "documentation" would be generous. Merlin had written most of it in High Enochian, and what little bits were in Elvish just seemed to be rants and raves about changes management had requested, and how wrong those changes were. The only system diagram was arranged into an Elder Sign. The most useful thing Renee found in her first pass were post-it notes with login credentials for the various bits.
Still, it was enough for Renee to start piecing together the system. The TPS report came from data managed by NNOSE, but the mainframe-based CLI interface had been abandoned circa 2002. At that point, someone had purchased an expensive mainframe integration package from Initech, which was basically a way of slapping XML/SOAP webservices on top of existing mainframe code. A few years later, someone had slapped together a VB.NET (framework version 1.0) UI which called into that webservice. The VB.NET program actually extracted and printed the report.
Great! Renee kind of knew Visual Basic, or at least could fake it. And it wasn't LUMPS. She started by testing the NNOSE XML interface- she sent a few SOAP requests manually and verified that she got correct output. Then she fired up the VB.NET app. It had a basic UI for setting filters and criteria, but once you hit "run report", everything-even the error messages-went to the printer. Worse, you couldn't tell it what printer to use. Every print went to one printer down in the reindeer stables.
Cue the testing cycle of "try running the report, ride the world's slowest elevator to the basement, walk through the reindeer stables, wait for the printer to finish running off its pages of error messages, drag the ream of paper back upstairs and try to match it to the code." Through this process, Renee learned a few important things. First, no matter how Christmas-y and magical reindeer were in spirit, in odor they were more Halloween Horror. Second, for some reason, there were SQL errors showing up in the stack traces. But everything was supposed to be webservices wrapped on mainframes, so why were there database calls?
Renee went back to Merlin's binders, and with the help of what little High Enochian she could remember from high school, Renee started to piece together what was happening.
Circa 2004, against Merlin's protests, there had been a project code-named RED. The goal of RED was to "upgrade" NNOSE to a full J2EE system with an Oracle backend. If Renee was reading the Enochian correctly, Merlin had banished the elf responsible for contracting the vendor to the Astral Plane, where they were surely captured by Githyanki.
"Well, that's got to be a violation of HR policies," Renee noted to herself.
The project never completed, but progressed to the point where the Oracle backend had been running in parallel with NNOSE ever since. That explained the SQL errors. The VB.NET app had functionality added to talk to the "upgraded" RED system. At some point in its history, the VB.NET app had been pointed to that backend, and had been querying it instead of the mainframe. Sure enough, when Renee logged into Oracle directly, most commands returned errors as something in this more-or-less unmaintained and unpatched Oracle instance was completely FUBAR.
Renee poked at the VB.NET app until she figured out how to resurrect the webservice-based data access layer. One more olfactory offense later, and Renee was holding a TPS report. Off it went to Santa, and off she went to give an after action to Hermie.
"Great work! You've saved Christmas!" Hermie said. "Shame about Merlin, though."
"What about Merlin?"
"Oh, the whole banishing poor Roger to be soulsucked on the Astral Plane. That is absolutely a firing offense, and we're going to have to let Merlin go. Don't worry, though. After this, I'm confident that you can fill in as our new NNOSE and LUMPS expert. You can think of it like a promotion, and maybe in a few years, it'll involve a salary increase!"
[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!