The Daily WTF

Carrol C just joined IniTech. They were hiring for someone who could help them tame their cloud costs. There’s a lot of money being spent on AWS. Management had bought in on the “it’s cheaper than on-prem”, and were starting to wonder why that promise wasn’t being fulfilled.After a little settling in period, Carrol had his credentials for their AWS environment, and started digging around just to familiarize himself with the infrastructure. This environment had started as an on-prem system that got migrated to the cloud, so the infrastructure was mostly a collection of virtual-machines using attached virtual disks- EBS- for storing data.And that was the first red flag. Each VM was using between 160–200GB of EBS. CPU usage was hovering at around 15%, and RAM was barely a blip, but the quantity of disk was getting a more than a little out of hand. Carrol scribbled a few notes on a post-it, and went down the hall to visit Merna’s cube. Merna was one of the engineers responsible for developing the application. “Hey, Merna, question for you: how big is our application code and deployable artifacts?”“Oh, depends on which instance you’re talking about, but the biggest weighs in at about a gig,” Merna said. “But we have some additional stuff installed on our VM image, so the VM image itself is about 4GB.”“And what does-” Carrol paused to check his note- “the WRPT–073 instance do?”“Oh, that’s just a simple webserver. Just generates some reports. Barely operates at capacity on even a Micro instance.”“So… why is the disk sized for 200GB?”Merna looked at Carrol like he had just asked the dumbest possible question. “Logs, obviously,” she said.“… you have 196ish gigs of logs?”Merna nodded. “We might need them.”“AWS has a log aggregator that doesn’t store the logs on EBS. You could just ship them there and use logrotate to trim your logs, and that would be way cheaper.”Merna shook her head. “You say that, but- oh, hold on.” Merna’s email bleeped at her: instance WRPT–073 was getting close to its disk capacity threshold. She quickly pulled up the AWS console and added another 10GB to the disk, before turning back to Carrol. “A lot of those accessory services have highly variable costs, and it makes it hard to predict your spend. Using cloud VMs is a much more consistent option. But if you feel so strongly about it, you could submit a request and we’ll evaluate it for a future release.”Carrol submitted a request, and also pinged his new boss. “I think I know a way we can manage costs.” For now, Merna and the other engineers just expand disks when they fill up. It remains to be seen if anything actually changes, but regardless, Carrol is prepared for an “interesting” time at IniTech. [Advertisement] Otter - Provision your servers automatically without ever needing to log-in to a command prompt. Get started today!

Vernice inherited a legacy web app. By "legacy" in this case, we mean "lots of jQuery." jQuery everywhere. Nested callbacks of HTTP requests, no separation of concerns at all, just an entire blob of spaghetti code that was left out on the counter and is now stuck together as a big blob of sauceless starch. And as for documentation? There isn't any. No technical documentation. No comments. The code didn't even pretend to be self-documenting.For the past few months, Vernice has been tasked with adding features. This generally meant that she'd find the code she thought was responsible for that section of the app, change something, see nothing happen, realize she was looking at the wrong module, try that three more times, finally find the actual code that governed that behavior, but as it turns out it had downstream dependents which broke.Adding a single textbox to a form was a week long process.So imagine Vernice's joy, when she opened a file and saw neat, cleanly formatted code, a compact function, and her text editor showed her the telltale color of comments. There were comments! Had she finally found the "good" part of the application?Then she read the code.

As we’ve explored recently, developers will often latch onto something they pick up in one language and carry it forward with them into others. Kerry still is working with the co-worker who has… odd ideas about how things should work. At is turns out, this developer is also a Highly Paid Consultant, which we just discussed yesterday.The latest problem Kerry found was in a display grid. It lists off a bunch of information linked to the user’s account, and each entry on the grid has a little plus sign on it to display further details. What, exactly, appears on that grid is tied to your account. It’s also worth noting that this service has a concept of corporate accounts- a “parent” account can administer entries for all their child accounts.When someone clicks that little plus sign, we need to fetch additional details, regardless of whether or not they’re a corporate parent account or not. So how exactly does the C# controller do this?

IniTech’s fashion division, IniGarment, launched their new accounting solution by hiring “best in class” highly-paid-consultants (HPCs). The system launched, collapsed under the weight of the first week of use, hardware was thrown at the problem in an absolute crisis panic, and the end result was that they had a perfectly serviceable accounting package that was overbudget and supported by expensive HPCs.It wasn’t sustainable. So out the HPCs went, and in came a pile of salaried employees. Jeff was one of those. His first six weeks at IniGarment were spent trying to understand the nest of stored procedures, quick hacks, and ugly choices. One of the first puzzles Jeff decided to look into was an invoice uploading step.They used a turn-key eCommerce package to handle sales, and the accounting system needed to generate invoices and upload them to the eCommerce package. This relatively simple task took 30 minutes to generate and upload a single invoice, and that was on a good day. On bad days, it could take nearly an hour. So Jeff set out to figure out why.In the middle of a 700+ line stored procedure, Jeff found the query which generated the report. The whole process required big piles of temporary tables (which, instead of being true temp tables, were created/dropped with each execution), and ugly group-bys and subqueries. Still, even with all that, it was just a SQL query, right?

I use Python a lot at work, and if you're doing anything vaguely data oriented, you want to use NumPy. I gave a talk about how much I love NumPy. It's one of the things that I automatically include in every requriements.txt because it's so goddamn useful.Lanny supports a product which uses NumPy, which is why he was surprised to find this block:

Adam S sends us a representative line which represents a mystery. It's a simple enough Java statement:

As the saying goes, there are two hard problems in computer science: naming things, cache expiration, and off-by-one errors. This little snipped of anonymously supplied PHP code highlights the first one of those challenges:

Odette’s company uses a popular video conferencing solution for browser use. In the base library, there’s a handy-dandy class called ObjectStorage, which implements an in-memory key/value store entirely in TypeScript/JavaScript.“Wait,” you ask, “isn’t a JavaScript in-memory, key/value store just… an object? A map, if you’re being pedantic?”Yes, of course, but have you thought about ways to make JavaScript’s maps/objects worse?

I’m starting to wonder if we need a “Representative API” category. There are just so many ways in which a developer can mess up basic tasks, like mapping a RESTful API to their backend.Today’s anonymous submitter was working with a REST API provided by a “world leading parcel” company. Everything started well. The documentation was thorough, contained links to example projects, and it came with a Swagger API doc. Based on that documentation and based on the Swagger doc, they went ahead and tried to CURL the API for a specific endpoint:

There's a very specific brand of bad code that I see from time to time, which I think of as "Oh, this poor person was permanently damaged by exposure to C." They're not writing C, but there's something about their "accent" which tells you: they learned C and didn't recover from the experience. Every reference variable can be treated like a pointer if you're insistent enough.There are other, similarly brain-breaking languages. COBOL. PL/SQL. VBA. Programmers learn the quirks of these languages, fail to understand them, and then start writing weirdly formal, structured code in languages that aren't supposed to work that way.Kerry has a co-worker like that. They're doing C#, and they include weird little ticks that are a clear sign of some kind of other langugae trauma. My guess is that it's a mix of PL/SQL and Visual Basic (not .NET), based on my experience with other people who have suffered through learning PL/SQL and VB as their first languages.We're going to take a look at some. In the interests of anonymization, I'm not going to include all of the code, which is 632 lines of tedium in one method, but I'm going to include the important highlights. Like, for starters, the message signature:

Pierre is a contractor who supports a variety of companies helping them manage data-science and analytics applications. Code, in these contexts, is written to answer specific questions, driven more by mathematics than program design. That’s a polite way of saying that the vast majority of the code is unreadable nonsense that no one can explain.One of Pierre’s clients has decided to migrate to that fancy new thing they’ve been hearing about, “the cloud”, and thus they have to move all of their applications to that cloud. This suite of applications involved a website with a pile of “gadgets” that users could string together to do their own data analysis. And one of those gadgets was quickly thrown together by an intern circa 2007.Now, no one expects an intern to write good code on their own. They need guidance. They need experience. They absolutely don’t need to be thrown into the deep end to whip up a reusable “gadget” that nobody thinks will ever see production.So, within that context, the intern did their best. While working for the company, the intern had learned a little bit about GNU Octave, which is what drove most of their analytics. The intern also knew some PHP, from dabbling in web development. Since this particular gadget needed to store some data, the intern brought in the other tool they knew: MySQL.Thus was born the Intern’s Gadget, which works roughly like this:

Object-oriented languages often let you implement your own exception classes to hook into their structured exception handling. This is, usually, a good thing: you create your own custom types for your various errors, which makes various exception states more clear. PebkacException is more useful than just Exception, and you can build catch blocks specific to your application/API needs.But there’s a dark side to this feature. People want to hook functionality into their exception objects. Instead of just using them as a signal to announce a failure state, they slap features into them, like exceptions which automatically self log.Muriel discovered this PHP SelfLoggingException object when trying to throw an exception caused the application to crash with a stack overflow.

Despite founding The Daily WTF more than fifteen years ago, I still find myself astonished and perplexed by the curious perversions in information technology that you all send in. These days, I spend most of my time doing "CEO of Inedo stuff", which means I don't get to code that much. And when I do, it's usually working with the beautiful, completely WTF- and bug-free code that our that our world-class engineers create.I mention this, because when I come across TDWTF-worthy code on my own, in the wild, it's a very special occasion. And today, I'm excited to share with you one of the worst pieces of code I've seen in a very long time: EnglishPluralizationServices.csAnyone even remotely familiar with the English language knows that pluralization is hard, and not exactly something that should be generalized in library... let alone Microsoft's most strategic programming asset of the past two decades. And yet, despite that:

YouTube, like any reasonable video service, offers closed captioning. They'll even throw machine learning at the problem, and autogenerate captions, though that is usually only good for comedy, rather than actual accesibility.Any closed captioning system will generally let you specify the colors of the captions as well as the actual text. YouTube is no exception to that. YouTube offers an online editor, but anyone serious about producing content is going to upload their own subtitle files, and up until recently, this could be done in an XML file which would allowed a lot of control over the styling of the captions.But XML isn't cool, so YouTube rolled out a new JSON format. Which broke everything. Specifically, instead of being able to set any hex triplet as your color, you could only set a relative handful, and any "invalid" triple would just turn to white.For some reason, whatever the user inputs isn't propogated to this as a hex triplet, like #FF0000, but instead as a hex value- 0xFF0000, or 16711680, in decimal. I'm not sure if that's on the "filling out the JSON" side, or if it's a conversion which happens elsewhere, but regardless: the front end needs to map a value like 16711680 to #FF0000. Well, a value like that, but not exactly that value, as we'll see.An anonymous submitter already trawled through the minified code and found the code block responsible, which they've helpfully de-minified for us:

Are we sick of of year rollover bugs yet? Well, let’s just agree that people aren’t sick of making these kinds of bugs.A long time ago, someone at Oleksandr’s company needed to write a Python script that shipped a 4-digit year to a system that only accepted 2-digit years. So 2010 needed to turn into 10.They could have used date formatting to format the date into a two digit year, but that involves understanding how to format dates. That just seems like too much overhead, when there’s already a perfectly good way to mangle a string.

What, exactly set of features divide a "text editor" from an IDE is a bit of a blurry line. Developers are not the sort to use static, unchangeable tools. They want configuration, they want plugins, they want quick access to a terminal, they want debugging support, and they'll bolt those features into just about anything.There's a fine line between an IDE that provides nice utility to the developer, and an IDE which is opinionated. I had the misfortune long ago to use the WebSphere IDE, which was essentially a repackaged version of early Eclipse bundled with highly opinionated plugins about how you were supposed to build a Java application. As Matt puts it: "An IDE is, to some, a high-functionality tool for developing applications, and to others a classic example of an Inner Platform."Matt is saddled with a vendor-specific IDE which falls solidly on the Inner Platform side of things. Like many developers, Matt has a laptop which he can lug to meetings, and a docking station with a large external monitor he can use for actual work.There's just one problem. Any time this vendor-specific IDE wants to pop up a dialog box, the dialog box always appears on the primary monitor. Now, Matt leaves his laptop screen set as the primary monitor, but he's doing his actual work on the large monitor, so this is a nuisance, to be certain. Ideally, dialogs should appear on whatever monitor the active window which triggered the event is on.This is a minor bug. Almost trivial. But it is a bug, and enough of a nuisance to Matt that he entered a bug ticket. He fired it off, and largely forgot about it over the next two weeks, until he got a reply.

There’s a simple rule about “enterprise” software: if the word “enterprise” is used in any way to describe the product, it’s a terrible product. Enterprise products are usually pretty special purpose and serve a well-capitalized but usually relatively small market. You aren’t going to sell licenses to millions of companies, but maybe tens of thousands. Often hundreds. There are some extremely niche enterprise products that have only two or three customers.Lots of money, combined with an actually small market in terms of customers, guarantees no real opportunity for competition. Couple that with the fact that each of your customers wants the off-the-shelf product you’re selling them to have every feature they need for their business case, you’re on a fast track to bloated software, inner platforms, and just general awfulness.Today, we’re not talking about Oracle, though. Jacek is in the healthcare industry. Years ago, his company decided to shove off their patient scheduling and billing to a Software-as-a-Service enterprise product. Integrating that into all of their other systems has kept the development teams busy ever since, and each successive revision of the API has created a boatload of new work.Currently, the API is on version six. V6 is notable because this is the first version which offers a REST/JSON API. V5 was a SOAP-based API.Here’s an example message you might send to their scheduling services:

Daniel didn't believe the rumor at first. Whenever his company chased after the hottest new business trends, they usually pursued the worst trends imaginable. But word was that this time, they'd seen fit to mimic Google’s fabled "20% Time."The invitation for a company-wide meeting soon landed in everyone's inbox, turning rumor into reality. For a moment, Daniel dared to dream about the various time-saving scripts he'd always wanted to write, and the applications he'd always wanted to re-implement from the ground up—all those things he'd been prevented from even thinking about as he'd been flung from one urgent project to the next. With 20% of his work time dedicated toward personal pursuits, all of that was set to change ...He forced himself not to get excited. As a hardened corporate veteran, Daniel had been down this road before. It wasn't a question of would this go wrong, but how would it go wrong.At the company meeting, which Daniel attended via conference call, a parade of high-level executive voices gushed about the new corporate policy, dubbed "Take-12." "From now on, everyone gets one hour each month to put toward personal work and development!"Twelve hours a year, out of the 2,000 that most people were expected to work. Daniel supposed "0.6% Time" didn't have quite the same ring to it.And how would the new policy be reconciled with the company's already Byzantine timesheet and internal billing system? Well, the company didn't even even attempt such a thing; there was simply no room in anyone's budget. Instead, someone had the bright idea of co-opting a different internal system for curating video playlists for online training courses. The idea was that each employee would create a new playlist of 12 hour-long videos, one for each month. The videos wouldn't actually be videos, just placeholders. By "watching" a "video," an employee would be logging that they'd spent one hour of time on personal development.The premise started off sketchy, and only got sketchier as difficulties were encountered during implementation. Daniel could only shake his head in astonishment as he read the User Guide detailing what he had to do:

There are certain things which you see in code that, at first glance, if you haven’t already learned better, look like they might almost be clever. One of those in any construct that starts with:switch(true) {…}It seems tempting at various points. Your cases can be boolean conditions now, but you can also collapse cases together, getting tricky with breaks to build complex logic. It’s more compact than a chain of ifs. It’s also almost always the wrong thing to do.Kasha stumbled across this while tracking down a bug:

We’re still in the early part of the year, and as little glitches show up from “sliding window” fixes to the Y2K bug, we’re seeing more and more little stories of other date rollover weirdness in our inbox.Like, for example, the Y2K15 bug, which Encore got to get surprised with. It feels like date issues are turning into a sports game franchise: new releases of the same thing every year.A long, long time ago, Encore’s company released a piece of industrial machinery with an embedded controller. It was so long ago and so embedded that things like floating point operations were a little to newfangled and expensive to execute, and memory was at an extreme premium.The engineer who originally designed the device had a clever solution to storing dates. One byte of EEPROM could be dedicated to storing the last two digits of the year. In RAM, a nibble- 4 bits- would then store an offset relative to that base year.Yes, this had Y2K issues, but that wasn’t really a concern at the time. It also had a rollover issue every 16 years. That also wasn’t really a concern, because it was attached to a giant machine which needed annual service to keep functioning properly. Every few years, the service tech could bring an EEPROM progammer device and flash the base year value in the EEPROM. And if someone missed 16 years worth of service calls, they probably had other problems.Time passed. Some customers did miss 16 years of service calls. Over time, new features got added. The control interface got an improved LCD. Bluetooth got attached. The networking stack changed. A reporting database got bundled with the product, so all the data being produced by the device could get aggregated and reported on. The way the software interacted with the hardware changed, and it meant that the hardware ran at a lower temperature and could go longer between service calls. But at its core, the chip and the software didn’t change all that much.In that time, there were also changeovers in the engineering team. People left the company, new engineers joined, documentation languished, never getting updated. Years might pass without anybody touching the software, then suddenly a flurry of customer requests that needed patched RIGHT NOW would come through, and anybody who vaguely understood the software got roped in to do the work, then shunted back off to other projects.On New Year’s Day, 2016, a deluge of tickets started coming in. Encore, as the last person to have touched the software, started picking them up. They all expressed the same problem: the date had rolled over to 2000. The reporting database was confused, the users were confused, and even if they tried to set the clock to 2016 manually, it would roll back from 2015 to 2000.Now, no one at the company, including Encore, actually knew about the date system in use at this point. The support manual did say that rollovers meant the device had gone 16 years without being properly serviced, but some of these customers had brand new devices, less than a year old. And customers with devices older than 16 years weren’t seeing this problem.Encore investigated, and picked apart how the date handling worked. That, itself, wasn’t the problem. It took a lot more investigation to track down the problem, including going back to the board schematics to trace how various hardware components were connected. After a few hair-on-fire weeks of crisis management, Encore pieced together the series of events as they were best able.Sometime after the year 2000, Bluetooth was added to the device. Something about how the Bluetooth module connected to the other components had broken the flasher-software that could update the base year. This meant that the devices had never had their base year set, and simply had a 0 value- 0x00, or the year 2000.Which meant, for the next 16 years, everything was fine. Techs went out, tried to flash the EEPROM, reset the clock to the correct date, and went about their business, never aware that they hadn’t actually done anything. But come 2016, all of these devices rolled back over to the year 2000.Encore was able to figure out a script to trick the system into adjusting the output to correct the base year issue, but it also meant many customers had database crammed with bad data that needed to be adjusted to correct the erroneous year.After this, Encore’s company released upgraded version of the system which contained a GPS receiver, so that it could set its date based on that, but a large number of their customers weren’t interested in the upgrade. Encore has already blocked off the first few weeks of 2032 in preparation. [Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

There’s always a hope that in the future, our code will be better. Eventually, we won’t be dealing with piles of krufty legacy code and unprepared programmers and business users who don’t understand how clicking works. It’s 2020: we officially live in the future. Things aren’t better.Duane works in Go, and has a piping hot “Representative Line” written in 2020. If, like me, you don’t quite know Go, it still looks pretty terrible at first glance:

As we've discussed in the past, video game code probably shouldn't be held to the standards of your average WTF: they're operating under wildly different constraints. So, for example, when a popular indie game open sources itself, and people find all sorts of horrors in the codebase: hey, the game shipped and made money. This isn't life or death stuff.It's a little different when you're building the engine. You're not just hacking together whatever you need to make your product work, but putting together a reusable platform to make other people's products work.Rich D, who previously shared some horrors he found in the Unreal engine, recently discovered that UnrealScript has a useful sounding JsonObject. Since Rich is thinking in terms of mods, being able to read/write JSON to handle mod configuration is useful, but anyone designing a game might have many good reasons to want JSON documents.The file starts promisingly with:

"So, let me get this straight," Derrick said. He closed his eyes and took a deep breath while massaging his temples before letting out an exasperated sigh. "Not a single person... in this entire organization... is taking ANY responsibility for Ropeway? No one is even willing to admit that they know anything about this application...?"The Operations team had grown accustomed to their new director's mannerisms and learned it's just better to stay silent and let Derrick think out loud. Afterall, no one envied his job or his idealistic quest for actual compliance. If had he been at the bank as long as his team had, Derrick would have learned that there's compliance... and then there's "compliance.""But we figured out that Ropeway somehow automatically transfers underwriting overrides from ISAC to AppPortal?" Derrick paused to collect his thoughts before a lightbulb went off. "Wait, wait. Those systems are both covered under our IBM Master Service Agreement, right? What did they say? Chris... did you reach out to our IBM liaison?""Well," Chris silently thanked everything good that Ropeway wasn't his problem. "IBM says that they have no idea. They said it's not in the scope of the MSA or any SOW, but they'd be happy to come out and—""Ab-so-lute-ly not," Derrick interrupted. He wasn't IBM's biggest fan, to put it mildly. "I've already eaten into next year's budget on this SSL initiative, and there's no way I'm gonna pay them just to tell me I have to pay them even more to fix what shouldn't even by my problem!"Derrick let out another sigh, rubbing his temples again. "All I want," he grimaced, "is for Ropeway to use HTTPS instead of HTTP. That's all! Fine... fine! Chris, let's just move the whole damn Ropeway server behind the proxy.""Roger that," Chris nodded, "We'll start prepping things for next week's maintenance window."There was a lot of risk to moving Ropeway. The Operations team knew how to keep it running – it was just a Windows Service application – but they had no way of knowing if the slightest change in the environment would break things. Moving the server behind the http-to-https proxy meant a new IP and a new subnet, and they had seen far too may "if (IP==10.10.22.30) production_env = true" traps to know they can't just move things without a test plan.But since no one on the business or Development side was willing to help, they were on their own and it'd be Derrick's head if ISAC or AppPortal stopped working once that maintenance window was over. But for the sake of actual compliance – not "compliance" – these were the risks Derrick was willing to take: SSL was simply non-negotiable.##"You'll never believe what I found on that Ropeway server," Chris said while popping into to Derrick's office. Actually, he knew that wasn't true; Derrick had come to expect the unbelievable, but Chris liked to prep Derrick nonetheless. Derrick took a deep breath and moved his hand towards his forehead."I found this." Chris plopped down a thick, tattered manila envelope that was covered in yellowed tape. "It was... um... taped to the server's chassis."Derrick was legitimately surprised and started riffling through the contents as Chris explained things. "So apparently Ropeway was built by this guy, Jody Dorchester, at Roman, uh, wait. Ronin Software or something.""And yeah," Chris continued as Derrick's eyes widened while he flipped through page-after-page-after page of documentation, "Jody apparently wrote all sorts of documentation... installation instructions, configuration instructions – and all the source code is on that enclosed CD-ROM."Derrick was speechless. "This," he stuttered, "is dated... March ...of 2000.""Yup," Chris jumped in nonchalantly, "but I took a shot in the dark here and sent Jody an email.""And...," Chris said, smiling. He handed Derrick another document and said, "here's his reply."

"For my sins," John writes, "I'm working on a SharePoint 2010 migration."This tells us that John has committed a lot of sins. But not as many as one of his coworkers.Since they were running a farm of SharePoint servers, they needed to know what was actually running, which was quite different from the documentation which told them what was supposed to be running. John's coworker did some googling, some copy-and-pasting, some minor revisions of their own, and produced this wad of PowerShell scripting which does produce the correct output.

Twenty years out, people have a hard time remembering that Y2K was an actual thing, an actual problem, and it was only solved because people recognized the danger well ahead of time, and invested time and effort into mitigating the worst of it. Disaster didn’t come to pass because people worked their butts off to avoid it.Gerald E was one of those people. He worked for a cellular provider as a customer service rep, providing technical support and designing the call-center scripts for providing that support. As 1999 cranked on, Gerald was pulled in to the Y2K team to start making support plans for the worst case scenarios.The first scenario? Handling calls when “all phone communication stopped working”. Gerald didn’t see much point in building a script for that scenario, but he gamely did his best to pad “we can’t answer the phones if they don’t ring” into a “script”.There were many other scenarios, though, and Gerald was plenty busy. Since he was in every meeting with the rest of the Y2K team, he got to watch their preparedness increase in real time, as different teams did their tests and went from red-to-green in the test results. A few weeks before the New Year, most everything was green.Y2K fell on a Saturday. As a final preparation, the Y2K team decided to do a final dry-run test, end-to-end, on Wednesday night. They already ran their own internal NTP server which every device on the network pulled from in one way or another, so it was easy to set the clock forward. They planned to set the clock so that at December 29th, 22:30 wall-clock time the time server would report January 1st, 00:00.The Y2K team gathered to watch their clock count down, and had plans to watch the changeover happen and then go party like it was 1999 while they still had time.At 22:29, all systems were green. At 22:30- when the time server triggered Y2K- the entire building went dark. There was no power. The backup generator didn’t kick on. The UPSes didn’t kick over. Elevator, Phones, HVAC, everything was down.No one had expected this catastrophic a failure. The incident room was on the 7th floor of the building. The server room was in the basement. Gerald, as the young and spry CSR was handed a flashlight and ended up spending the next few hours as the runner, relaying information between the incident room and the server room.In the wee hours of the morning, and after Gerald got his cardio for the next year, the underlying problem became clear. The IT team had a list of IT assets. They had triaged them all, prioritized their testing, and tested everything.What no one had thought to do was inventory the assets managed by the building services team. Those assets included a bunch of industrial control systems which managed little things, like the building’s power system. Nothing from building services had ended up in their test plan. The backup generator detected the absence of power and kicked on- but the building’s failure meant that the breakers tripped and refused to let that power get where it was needed. Similar issues foiled their large-scale UPS- they could only get the servers powered up by plugging them directly into battery backups.It was well into the morning on December 30th when they started scrambling to solve the problem. Folks were called back from vacation, electricians were called in and paid exorbitant overtime. It was an all-hands push to get the building wired up in such a way that it wouldn’t just shut down.It was a straight crunch all the way until New Year’s Eve, but when the clock hit midnight, nothing happened.

As a general rule, if you find yourself writing an extension system for your application, stop and do something else. It's almost always in the case of YAGNI: you ain't gonna need it.George is a "highly paid consultant", and considers himself one of the "good ones": he delivers well tested, well documented, and clean code to his clients. His peer, Gracie on the other hand… is a more typical representative of the HPC class.George and Gracie found themselves with a problem: based on the contents of a configuration file, they needed to decide what code to execute. Now, you might be thinking that some kind of conditional statement or maybe some sort of object-oriented inheritance thing would do the job.There were five different code paths, and no one really expected to see those code paths change significantly. Gracie, who was identified as "the architect" on the responsibility matrix for the project, didn't want to write five different ways to do a similar task, so instead, she wrote one way to do all those tasks.Here's the YAML configuration file that her efforts produced:

Kaylee E made an "oops" and checked in a unit test with a bug in it which caused the test to fail. She didn't notice right away, and thus the commit hit their CI pipeline and was automatically pulled by the build server. She assumed that when she checked the logs she'd see the error, but she didn't. The build completed, and Tests (0/0) ran successfully.Now, Kaylee was new to the codebase, and since she'd been doing small changes, she'd simply written and run tests around explicitly the functionality she was testing. She hadn't yet done a full test run locally, so that was her next step. From there, it was easy to see why the build server didn't automatically run tests.

As we plow into the holiday season, it’s important to remember that each submission- each bit of bad code, each horror story, each personal confession- is its own little gift to us. And, when you write a bit of bad code, you can think of it as a gift for whoever follows you.Georgeanna recently opened a gift. She was wondering how their logging layer managed its configuration. She assumed that it would just read it from the config file, but when she tried to change where the logging file got written, say, to report.log, it would turn into report.log.staging.log.It wasn’t hard to figure out why:

Our anonymous submitter—we'll call him Russell—was a senior engineer supporting an equally anonymous web service that was used by his company's desktop software for returning required data. Russell had a habit of monitoring the service's performance each day, always on the lookout for trouble. One fateful morning, the anomalies piled on thick.Over the past 24 hours, the host server's average response time had halved, and yet the service was also suddenly dealing with four times as many requests as usual. Average CPU and memory usage on the server had doubled, as had the load on the Oracle host. Even stranger, there was no increase in server errors.Russell couldn't imagine what might've happened, as no changes had been deployed. However, his product team had recently committed to reducing average server response time. It was possible that someone else had modified an upstream service or some database queries. He emailed the rest of the team and other teams he worked closely with, detailing what he'd seen and asking whether anyone had any pertinent information.The response from the engineers was basically, Hmm, odd. No, we didn't change anything. The response from the product architects really shouldn't have surprised Russell, given he'd been working in enterprise for nearly 20 years. The reply-all frenzy can be summed up as, You mean we've already fulfilled our commitment to reduce average response time?! LET'S FIRE OFF A SELF-CONGRATULATORY COMPANY-WIDE EMAIL!!!Upon seeing this, Russell immediately replied: Hold on, let's try to find out what's happening here first.Unfortunately, he was too late to stop the announcement, but that didn't stop him from investigating further. He remembered that their default monitoring of server errors filtered out 404s. Upon turning off that filter, he found that the number of 404s thrown by the server roughly matched the number of additional requests. Previously, average response time had been around 100ms; at present, it was about 45ms. This "triumph" hid the fact that the numerous 404s were processed in about 10ms each, while the non-404 requests were processed in about 150ms each—50% slower than usual. In other words, the web service's performance had been seriously degraded.Russell dug further to figure out who was performing this low-key DDoS attack. The requests were authenticated, so he knew the calls were coming from inside the house. He managed to trace them to another product within his company. This product had to make a request to his web service in about 1% of their sessions, but that considerably slowed down their handling of those particular sessions. As a result, someone had modified the product to fire off an asynchronous request to Russell's service for every session, simply ignoring the response if it was a 404.Russell emailed his findings to his team, but received no reply. Feeling bold, he directly contacted the project manager of the offending product. This led to the biggest WTF of all: the PM apologized and got the change rolled back right away. By the next day, everything was back to normal—but the product architects were angry over the embarrassment caused by their own premature celebration. They were likely also miffed about being forced to find real ways of improving average server response time. Their misplaced ire led to Russell being fired a short time later.However, our story has a happy ending. The super-responsive product team hired Russell back on after a couple of months, with a 25% pay raise. He retained seniority, and was allowed to keep his former benefits as well as his severance package. In the end, the forces that'd sought to be rid of him had only succeeded in giving him a highly-paid vacation. [Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Jarad was still recovering from his encounter with Intelligenuity’s most “brillant” programmer, Keisha, when a new hire, Aaron, showed up at Jarad’s office.The large project that dominated their timelines remained their efforts to migrate from .NET to Java, but Aaron was hired to keep the .NET side of things on track, handling bugs, new features that were desperately needed, and just general maintenance. It was made emphatically clear by the project managers that hiring more .NET developers was not an admission that the conversion to Java had failed, but would “free up resources” to better focus on the Java side of things.Aaron moved fast to establish himself. He scheduled a presentation in the first week. He was vague about what, exactly, the presentation was about ahead of time. So, when the lights came down and the projector lit up, everyone was a bit surprised to see their .NET code in his slides.“This,” he explained, “is our application code. I wanted to give you a walk through the code, so we all as a team have a better understanding.”Jarad and his co-workers exchanged glances, silently wondering if this was for real. Was Aaron really about to explain the code they had written to them?“This line here,” Aaron said, pointing to a for loop, “is an interesting construct. It will repeat the code which follows to be repeated once for each element in the array.” A few slides later, highlighting a line which read, x = new AccountModel(), Aaron explained. “This creates an instance of an account model object. The instance is of the class, while the class defines what is common across all objects.”That hour long meeting was one of the longest hours of Jarad’s life. It was a perfect storm of tedium, insult, and incompetence.Afterwards, Jarad grabbed his manager, Regine. “Like, do you think Aaron is going to actually be a good fit?”“Oh, I’m sure he’ll be fine. Look how well he understands our code already!”That laid out the pattern of working with Aaron. During one team meeting, the team got sidetracked discussing the best approach to managing a very specific exception in a very specific section of their code. Fifteen minutes after the meeting, Aaron followed up with an email: “Re: Exception Handling”, which consisted of a bad paraphrase of the Execption class documentation from the MSDN site. Another day, during another meeting, someone mentioned concurrency, so Aaron followed up with an email that broadly plagiarized a Stack Overflow post describing the ProcessThread object.And, on each one of those emails, Regine and several other project managers were CCed. The result was that the management team felt that Aaron was a great communicator, who constantly was adding value to the team. He was a mentor. An asset. The kind of person that should be invited to every one of the project management meetings, because he was extremely technical but also the kind of communicator and go-getter that had management written all over him.Among the developers, Aaron’s commits were a running joke. He submitted non-working code, code that violated every standard practice and styleguide entry they used, code with out tests, code with tests that would pass no matter what happened, code that didn’t compile, and code that was clearly copy/pasted from a tutorial without bothering to try and fix the indentation.It was no surprise then, that a few months later, Aaron announced that he was now a “System Architect”, a role that did not actually exist in their org-chart, but Aaron assured them meant he could tell them how to write software. Jarad went to Regine, along with a few other developers, and raised their concerns. Specifically: Aaron had invented a new job role and was claiming authority he didn’t have, he didn’t have the seniority for a promotion at this time, he didn’t actually know what he was doing, and he was killing team morale.“Are you familiar with the crab mentality?” Regine asked. “I’m concerned that you’re being poor team players and a negative influence. You should be happy for Aaron’s success, because it reflects on how good our team is!”Jarad and the rest of the team soon discovered that Regine was right. Now that Aaron was a “System Architect” he was too busy building presentations, emailing barely comprehensible and often inaccurate summaries of documentation, and scheduling meetings to actually write any code. Team performance improved, and it was trivial to configure one’s inbox to spam Aaron’s messages.Aaron’s “communication style” kept getting him scheduled to do more presentations where he could explain simple programming concepts to different layers of management. The general consensus was that they didn’t understand what he was talking about, but he must be very smart to talk about it with a PowerPoint deck.After their next release of their .NET product, Aaron scheduled a meeting with some of the upper tier management to review the project. He once again dazzled them with his explanation of the difference between an object and a class, with a brief foray into the difference between reference and value types, and then followed up with an email, thanking them all for their time.On this email, he CCed the VP of the company.The VP of the company was also one of the founders, and was a deeply technical person. She never related her reasoning to anyone, but based on Aaron’s email, she scheduled a meeting with him. It was no trick finding out that the meeting was going to take place: Aaron made sure to let everyone on the team know. “I have to block off everything from 3PM on Thursday, because I have a meeting with the VP.” “Can we table that? It’s probably best if we discuss after my meeting with the VP.” “I’ll be back later, it’s time for my meeting with the VP.”No one knows exactly what happened in that meeting. What was said or done is between Aaron and the VP. But 45 minutes later, both Aaron and the VP walked onto the developers’ floor. Aaron was watching his shoes, and the VP was staring daggers at the back of his neck. She marched Aaron into Regine’s office, and closed the door. For the next twenty minutes, the VP vented her frustration. When her voice got raised, words like “enabling” and “incompetence” and “inappropriate” and “hiring practices” leaked out.The VP stormed back out, leaving Regine and Aaron to discuss Aaron’s severance. That was the last day anyone saw Aaron.Well, until Jarad started thinking about attending a local tech conference. Aaron, as it turns out, will be one of the speakers, discussing some “cutting edge” .NET topics. [Advertisement] ProGet can centralize your organization's software applications and components to provide uniform access to developers and servers. Check it out!

Sometimes, it feels like any programming question you might have has a thread on StackOverflow. It might not have an answer, but it’s probably there. Between that, online guidebooks, tools with decent documentation, YouTube programming tutorials there are a lot of great ways to learn how to solve any given programming task.Andreas R had a programming task. Specifically, Andreas wanted to create sortable tables that worked like those on MediaWiki sites. A quick google for “sort html table” turned up a source which offered… this.

Java date-time handling was notoriously bad for the vast majority of Java's lifetime. It was so bad that a third party library, Joda-Time, was the defacto standard for Java date processing until finally, in Java 8, the features, functionality, and design of Joda-Time were adopted into Java. JSR-310 added refinements to conventional datetime objects, like Timestamps and LocalDates, but also added useful classes like Instant (an immutable instant in time) and DateTimeFormatters that had a conventional and flexible API for doing date formatting and parsing.Since JSR-310, it's easy to write good date handling code in Java.That, of course, doesn't mean that you can't still write terrible date handling code. Normally, you'd expect your bad date handling code to take the form of one of the standard badnesses: write your own string mangler, insist on using the legacy libraries, homebrew a stacked up library of edge cases and ugly code and weird misunderstandings of the calendar.Brendan sends us an example where they manage to use the new APIs in a head-scratching fashion.

Seth S offers us something new: a representative line of Ada. We don’t get much of that, and Ada isn’t a particularly popular language, but Seth assures us that it is “unfairly maligned”.Since 1995, Ada has been an object oriented language, and offers a standard library, strong types, a message-passing approach to communicating with objects (which migrated into Objective-C but generally doesn’t show up very often elsewhere). It’s a fine, if less-used language, and I honestly can’t say I’ve heard much maligning it (though I’ve never actually heard of anyone using it either…).Regardless, what we can malign is some bad code. Since the earliest versions of Ada, if you wanted to find the absolute value of a variable, you’d write an expression like this:

Nulls cause problems. Usually, they’re not big problems, but if a field might have a value- or none at all- we have to be careful with how we handle it.Languages like C# have added Nullable types, which wrap around those problems. But sometimes, you need to cross a boundary between systems. When you send the C# data to JSON, how do you want to represent null values?You might just send nulls. That’s fine and logical. You might just leave out the null keys (technically sending undefined). Also fine and also logical, as long as those sorts of variations are communicated by your schema.If you’re Jackie’s co-worker, you might decide that they should just be empty strings. This is a bad choice- if a field is an integer, but it doesn’t have a value, it suddenly turns into a string? But hey, you can document this too, and essentially treat the field as a union type. It’s ugly, but workable.Now, they use the Newtonsoft serializer to build their JSON, which is flexible and extensible, and with a little munging, can be tricked into converting nulls to strings. It’s a little bit of code, but a perfectly manageable thing, if you really want to do this.Jackie’s co-worker felt that it was too much code.This is what they did: