Best of…: Best of 2021: It's a Gift
Per tradition, we're taking the week before the new year as a chance to review some of our favorites from this year. We open with this one from way back in January. Consider it... a gift.
Tyra was standing around the coffee maker with her co-workers when their phones all dinged with an email from management.
Edgar is no longer employed at Initech. If you see him on the property, for any reason, please alert security.
"Well, that's about time," Tyra said.
They had all been expecting an email like that. Edgar had been having serious conflicts with management. The team had been expanding recently, and along with the expansion, new processes and new tooling were coming online. Edgar hated that. He hated having new co-workers who didn't know the codebase as intimately as he did. "My technical knowledge is a gift!" He hated that they were moving to a CI pipeline that had more signoffs and removed his control over development. "My ability to react quickly to needed changes is a gift!" He hated that management- and fellow developers- were requesting more code coverage in their tests. "I write good code the first time, because I've got a gift for programming!"
These conflicts escalated, never quite to screaming, but voices were definitely raised. "You're all getting in the way," was a common refrain from Edgar, whether it was to his new peers or to management or to the janitor who was taking too long to clean the restroom. It seemed like everyone knew Edgar was going to get fired but Edgar.
Six months later, the team was humming along nicely. Pretty much no one thought about Edgar, except maybe to regale newbies with a tale of the co-worker from hell. One day, Tyra finished a requirement, ensured all the tests were green in their CI server, and then submitted a pull request. One of her peers reviewed the request, merged it, and pushed it along to their CD pipeline.
Fortunately for them, part of the CD step was to run the tests again; one of the tests failed. The failing test was not anything related to any of the changes in Tyra's PR. In fact, the previous commit passed the unit test fine, and the two versions were exactly the same in source control.
Tyra and her peers dug in, trying to see what might have changed in the CD environment, or what was maybe wrong about the test. Before long, they were digging through the CD pipeline scripts themselves. They hadn't been modified in months, but was there maybe a bad assumption somewhere? Something time based?
No. As it turned out, after many, many hours of debugging, there was an "extra" few lines in one of the shell scripts. It would randomly select one of the Python files in the commit, and a small percentage of the time, it would choose a random line in the file, and on that line replace the spaces with tabs. Since whitespace is syntactically significant in Python that created output which failed with an IndentationError.
A quick blame confirmed that Edgar had left them that little gift. As for how it had gone unnoticed for so long? Well, for starters, he had left during that awkward transition period when they were migrating new tools. The standard code-review/peer-review processes weren't fully settled, so he was able to sneak in the commit. The probability that it would tamper with a file was very low, and it wouldn't repeat itself on the next build.
It was hard to say how many times this had caused a problem. If a developer saw the unit test fail after accepting the PR, they may have just triggered a fresh build manually. But, more menacing, they didn't have 100% unit test coverage, and there were some older Python files (mostly written by Edgar) which had no unit tests at all. How many times might they have pushed broken files to production, only to have mysterious failures?
In the end, Edgar's last "gift" to the team was the need to audit their entire CI/CD pipeline to see if he left any more little "surprises".
[Advertisement] Otter - Provision your servers automatically without ever needing to log-in to a command prompt. Get started today!