Thumbnail 1559656
thumbnail
Large (256x256)

Articles

Lose Some Padding
Flat-file style databases were designed to fit the constraints of the systems they were running on. You specify your schema in terms of "how many characters in a file we use to store this data", meaning something like this: JOHNSMITH12343rdStAnytown PA12345 is read in my knowing that the first name field is 8 characters wide, the last name field is 8 characters wide, the street number is 4 digits, and so on.It's also a terrible schema, and woe to anyone with a long name. But many a mainframe had a similar schema.Now, let's think about maintenance here. What happens when we also want to store a middle initial? We've created for ourselves a problem. Somehow, I have to insert a character into every row, which basically means making a new table with a new schema, copying every record out of it and updating it to use the new schema. I can't just ALTER TABLE like an RDBMS. And worse, every piece of software that touches the table also needs to be updated. On a large legacy system, a simple task like "add a field to our database" could take weeks of developer time, and depending on the software, be a high risk operation.Which is why the smart developer, when working with flat files, includes padding. Maybe my schema for an address record looks more like this: JOHNSMITH12343rdStAnytown PA12345. That's 16 characters of padding at the end of the file. Now somebody says that I need to store a middle initial, I can just shrink the padding by one and add a middle initial field, like so: JOHNSMITH12343rdStAnytown PA12345QIs this elegant? No. But it works. I haven't changed the length of the row at all, so I don't need to move data around. Software modules only need to be updated if they care about what's in the middle initial field; if they're out of date, they just think there's a "Q" in the padding, and don't care.In real-world applications, instead of putting all the padding at the end, you'd usually put the padding in a few spots in the middle of the table. Any time you need a new column, you just steal a few characters from padding. Sure, someday you'll run out of padding, or at least out of padding blocks big enough for your new field, and then you'll have to do the hard work of shuffling data around. But in practice, you can get very far without that happening.Which brings us to Brenda's adventure. Her team supports an IBM mainframe storing data in VSAM flat files. In other words, they've been doing the sort of thing I just talked about for many, many years.Of course, in the modern era, you can't just leave your data sitting in an mainframe. Even if the mainframe is the source of truth, you want to be able to report on it and connect it with your other data systems. You need to, somehow, get the data into a modern RDBMS.So the company hired a bunch of developers to write an extract-transform-load process, which pulls the data out of the mainframe. The mainframe team handed them a "copybook" for the flat file, which described the structure, and the ETL devs went to work.And maybe those ETL devs didn't understand the importance of padding. Maybe they just missed the padding. Whatever it was, there were several places where the data was structured like SOME_USEFUL_FIELD PADDING PADDING PADDING SOME_OTHER_FIELD, and they opted to split it like so: SOME_USEFUL_FIELD PADDING PAD, DING PADDING SOME_OTHER_FIELD.When they released this process, it was fine. The padding characters got stripped before displaying, so the users never saw them. They were stored in the database, though, so when someone tried to reconstruct the data in a way that was compatible with the flat files, you could just concatenate the columns together and get a valid result.It was fine- until it wasn't. The ETL devs, bless their hearts, only tested against the production mainframe. And why not, they were doing read only operations, what's the harm? Had they tested against the development mainframe, they would have seen new features in flight, features which consumed some of that padding, and realized that they should have paid closer attention to the copybook.But instead, the test cases all passed. The software was, as far as the project managers and ETL developers could tell, working perfectly. So it was accepted, released to production, and running for a few weeks before the mainframe released its features. Those features then ruined all the beautiful reports with extraneous data.And since the ETL devs were on contract, any request to have them rework it under the original contract was met with a stern "Works as designed". Instead of paying the contractors to come back and rework the system, the mainframe devs instead were tasked with finding different padding fields they could use, padding fields which wouldn't end up ruining any reports management liked to see. [Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!
CodeSOD: Always Take the Option
Frequent submitter Capybara James sends us this simple snippet, which highlights that even when you have the lovely convenience of Optional types, you can use them wrong.
CodeSOD: Connection State
Frederick A sends us a bit of null checking code, and offers us a better solution.
A More Civilized Age
Greta (previously) sends us more updates from her "Ancient Development Environment".An important task an IDE must do is report build errors to its users. Arguably, that's one of the most important parts. I wouldn't know, I insist on building from the CLI all the time, because IDEs confuse and frighten me. I recognize I'm the weird one here, who is more comfortable in GDB than in a GUI debugger, but this isn't about me, it's about the IDE Greta is using.It needs to display an error. Why does it need to display an error? Well, Greta hasn't figured that out yet. The error I'm about to show you doesn't really explain what happened or why or give any hint as to what needs to be done to fix it. To make matters more confusing, it doesn't happen consistently, so simply re-running the build could potentially fix it.None of that is why we're here, though. What makes this a WTF is how the error is displayed:Greta shares her bullet points about what she hates about this:
CodeSOD: Public Private Partnership
Eric O was trawling through an API for handling concurrency, and found this little mismatch between the comment and the definition:
Branching Paths
"You submitted a pull request."Indika was, in fact, reviewing the comments she'd gotten on that very same pull request, when her boss, Bill, walked up behind her. What she didn't understand is why Bill said it like it was an accusation."Yes?" she replied."Okay, well, we don't do that here. You're new, so I'll let it slide, but please review the developer guide."Well, Indika had reviewed the developer guide, or at least thought she had. As it turned out, there was the official, company wide developer guide. That's the one she'd read. But Bill maintained his own, for his team. He hadn't ever told her about it, but apparently assumed she'd have the oracular blessings of Apollo and find it by herself.It had this to say:
CodeSOD: Never Eating the Cookie
Maciej works as a freelancer, and that frequently means picking up old PHP code that nobody wants to support.One project had been lingering for ages with key features missing. Specifically, it was supposed to make HTTP requests to other services on an interval, and use that to populate its data. "The old dev tried, but never got it working." It was Maciej's turn to give it a shot.
The State of Ticketing
Developing software can't simply be done with a text editor and a compiler. There are a variety of other tools we have to bring to bear that support our efforts and keep the team organized, like say, source control.There are certain tools we all have to use that I would argue, nobody has actually make a version that's any good. Build tooling is one of my go-to examples: there are no good build systems, only build systems that are good enough for this task.Another is ticket/task management. In fact, I'd go so far as to say, there are no good ticket management tools. Amongst the not good tools, I'd put Jira as one of the not goodest of all.What makes Jira attractive to companies is the same thing that makes it miserable, and the thing that infects any "enterprise" software platform and turns it into garbage: it has all the features and expect you to build your own workflows with it. You don't merely use Jira, you have to program your own interfaces in Jira to get your workflow into the system. And if you have the misfortune to have a project manager who thinks they're more technical than they are, they'll endlessly spin up new views, new workflows, and rearrange how the work is tracked in lieu of actually working.I've been on that team.One of Jira's features is the ability to describe the ticket workflow: the state machine that describes your process from the initial entry of the ticket all the way down to released software or project completion. This includes routing, so that as one team member does their part of the work, it automatically goes to someone else to do the next portion of the work.Which brings us to Klinsten. They were working on a new team, and wanted to change the ticket status from its current status to whatever came next in the workflow. So they looked at the workflow.These are two different versions of the same workflow, one with transition labels added, which as you can see, does nothing to clarify the workflow. That it's a mix of Dutch and English doesn't help matters.The purpose of this workflow is to help the team understand how to sequence and organize their work. But this workflow has so many states and so many transitions, it fails at this goal. Looking at it makes me just want to gesloten my browser tab, because this user isn't accepting any of this. [Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!
CodeSOD: Back to the Lab
Matlab is special. Scientists and researchers love it. Programmers hate it, and not just because it uses 1-based arrays. I've worked on a number of projects where the task was "take this Matlab code and convert it to C so we can run it on an embedded CPU". Somehow, in that process, I've avoided learning much about Matlab.Andre works on a team that uses Matlab to manage experimental scenarios. They wanted to do a simple task: generate a set of participant-specific images, store them in a database, and reference them later. Somewhere in the intersection of the database product they were using, the Matlab license they had, and other constraints, they discovered that there simply was no good way to do this.Enter "Jude". Jude said, "Don't worry about it, I can hack something together."I present the code in its entirety, but don't ask me to explain it. Instead, read the comments.
Representative Line: We All Register This
Today's maybe more of a "representative data sheet entry" than anything else.Every developer has the experience of reading the documentation. If you've been at this for some time, you've probably read bad documentation. Documentation that is incomplete, inaccurate, or otherwise flawed. Or, my personal favorite, the brief time where Oracle tried to put all of its documentation into an Adobe Flex site (aka, a Flash application, not a real web app). That one had fun bonus features, like "breaking copy and paste" and "preventing you from deep linking to a piece of the documentation".But software documentation has got nothing on bad data sheets. When you buy an integrated chip from a vendor, whether it's a microcontroller that'll run your code, a sensor you're trying to get data from, you're at the mercy of the datasheet for understanding how it works. Sometimes, even finding an English language datasheet can be a challenge. The more complex the chip you're trying to interact with, the more complex the datasheet needs to be, and at a certain point, a lot of vendors say, "meh, you'll figure it out." I've had chips where the datasheet and reality disagreed about what registers were available, which often means that core functions of the chip require twiddling undocumented registers. For more fun, they sometimes lie about which pins on the chip do which thing, including mislabeling which pins handle power. There's nothing more fun than the tiny little "pop" of a chip dying when you throw 5V power onto a pin that's actually ground.Now, there are some vendors, and some products, where the datasheets are pretty solid. This isn't a universal problem, but when you're working in an embedded space, "cheapest" is frequently the main criteria for picking components, and "cheapest" means "worst documented".Which brings us to Jarek's recent experience going through a data sheet. The chip in question had a "fantastic feature" that would change how debugging worked, which was for "super users" to enable by setting a register.
Representative Line: Both Ways Bug Me
There are many cases where some sort of debugging block sneaks by, especially cases where we see preprocessors or templates working, which leave us with nonsense like if (true == false) running in production. But Codemonkey found a new twist on that sort of thing, in a SQL query being run in production.
CodeSOD: Lock 'Em Dead
Kevin sends us an exception handler from C++. Let's see if we can spot what's going wrong:
CodeSOD: The Big Family
Some time ago, Charles shared with us some awful PHP, aka the most common sort. Today's code sample is maybe a little too big to sum up, but I'll let Charles take a crack at it.
Representative Line: So Much Room
Today's representative comment ran out of room.
What You Measure
Rachel joined a new team which was proudly "metrics driven". When she first met with her boss, Zane, he explained his thinking."We need to be data-driven to make good decisions, right? We're a manufacturing company. We make widgets. At the end of the day, we need to make the most widgets for the lowest cost of goods sold. So we track that, and that feeds into every decision."The team oversaw an automated production line, which meant the software was a mix of robotics, embedded firmware, high-level web based monitoring tools, and thickets of dreaded PLC code. And because you can't build an entire factory for test purposes, they only way they could test real-world scales with real-world data was to roll changes out to production. They could simulate, they could run tests on subsets of the system, but a change in the production line software couldn't truly be validated until it rolled out into the real world.Rachel's first task on the new team involved making some changes to their metrics dashboard. It was viewed as a good way to get her feet wet with the new team. As it turned out, the metrics dashboard was a Google Sheet, with a complex series of formulas that involved multi-level INDEX functions- essentially querying the spreadsheets like they were a database. Why not use an actual database? Oh, they did - six actually - but the company obeyed Remy's Law of Requirements Gathering: "no matter what the requirements the users ask for, what they really wanted was Excel". The database data was pulled into the spreadsheet for reporting.Now, a complicated sheet pulling in data from not one, but six different databases, they must have a pretty complex model to explain how changes to their software would impact productivity. And since they needed to model the software to make predictions about how it'd behave in production, that model must be extremely useful.Of course it wasn't. The only metrics they tracked were output metrics, variations on "widgets produced per unit time". There were some performance metrics, so you could maybe potentially identify "oh, our overall throughput dropped because unit 5 became a bottleneck and started taking 1.5 extra seconds per widget", but nothing that actually helped you understand how the complex system made decisions. Or even why unit 5 was taking longer.For example, there was an automated quality control scanner. It examined widgets as they came off the line, and rejected defective ones based on a computer vision algorithm. Did that subsystem record why it rejected a widget? No, it did not. The CV model was able to tag widgets with a defect category based on what it saw, but that information didn't get recorded anywhere. In fact, it didn't even record how many widgets got rejected. The only way to know was to have an operator on the assembly line count widgets in the bin manually. Since that ate up a bunch of an operator's time, it never happened unless the developers begged for it. And since the operator still couldn't answer the question "why was this widget rejected", it wasn't all that useful anyway.Every change to the software was scored against the overall output metrics. This meant that when Rachel was ready to push out her first software change, something that would record how many widgets were rejected and why, whether or not it could be deployed was dependent on seeing the change improve, or at least not regress, the widgets-over-time scores. But the widgets-over-time were a noisy metric; it varied based on which operators were working any given shift, or based on supply chain constraints. Or sometimes, based on when one of the machines was last calibrated- theoretically something that happened on a set schedule, but really was up to the operators. This meant the first three times Rachel rolled her code out for a test run, the metrics regressed. Nothing she changed should have impacted the metrics, but the metrics regressed due to environmental issues.This meant making a simple change could take weeks, because you could only do final validation on the real system, which means you had to mark off a block of time for a test run, you could only run a handful of tests a day, and if metrics regressed you had to account for that before you could release the software for actual production use.Over the first few months, Rachel added instrumentation to the code. Anything along the way to generating an output widget, she recorded. The hope was that once they had enough data, they could build a useful model of the system. Unfortunately, Zane had other ideas."So, you haven't improved our metrics," Zane said. "Which, I remind you, we're a metrics driven organization. Every change needs to improve our metrics.""Sure, but I'm gathering more data so we have a better idea of what makes our metrics tick. We don't know why our system does some of the things it does, because we don't record any logging about the decisions it makes.""Right, but we already gather the key metrics.""But you don't gather the data that tells you why those metrics are what they are!""Sure," Zane said. "But those aren't our key metrics."That, unfortunately for Rachel, was where things landed. Understanding their complex system was a low priority. Pushing top-level metrics without understanding what fed into them, that was the priority. That didn't mean Rachel was powerless: any time she made a change that she thought might help the top level metrics, she also made sure to add instrumentation that explained how that change behaved. It was the compromise that kept Zane happy: she released features that impacted the top-level metrics, but she also made the system more observable. [Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!
CodeSOD: Heating Up
A common option for retrofitting heating and cooling into older homes is a mini-split, frequently tied to a heat pump. They're (relatively) cheap to install, energy efficient, and can be added without substantial modifications to the home. They also, annoyingly, are mostly controlled via IR remotes, making them challenging to wire up to home automation or even a household thermostat.People have made solutions, and today's code comes from one of those solutions. Which, I want to stress, this code comes from an open source project for home automation, so it's not the code that's wrong, here. At first I thought it was, and had a moment of, "I'm not going to pick on some hobby project," but then I realised the hobby project points at a deeper issue.
Best of…: Classic WTF: A Dumbain Specific Language
CodeSOD: Asynchronous Directories
Eri has a mix of a "true confession" and a "wait, really?" today.The programming language Vala bills itself as a C# like language that compiles into something pretty close to C performance, designed specifically for writing code against Gnome and its associated libraries.One of the C#-isms in brings in is async/await type semantics. You can yield someAsyncFunction(), which returns control to the caller, allowing it to proceed until the yielded function returns an actual value.Because it has asynchronous functions, many library functions for handling I/O are already async. So you can make_directory_async, which yields control so you can keep executing while waiting for the filesystem to make your directory.There are also synchronous versions of those methods. And then there's create_directory_with_parents, which will create a chain of directories for you. That's the synchronous version, and Vala's core library has decided not to provide an asynchronous version of it, which is my "wait, really?" I suspect it's really about the race conditions involved and the risks of things going wrong while doing it asynchronously; all solvable problems, but tricky ones to solve.But it's the problem Eri had, and this is their solution:
A Bit of DNS
I'm not a DNS person, in that I appreciate that it exists but am not up on the inner workings. It solves a lot of problems with dark magic I don't fully understand, and fortunately don't need to.But Lucio noticed something that I do think is interesting, within the scope of the CAA record type.The CAA record started with RFC6844, which was obsoleted by RFC8659. Both RFCs lay out the same core idea: you can add a CAA record to your DNS entries to say, "hey, this domain over here is allowed to issue certificates for me". That's the sort of thing that enables LetsEncrypt to hand out certs, and is an important part of why we can run HTTPS everywhere these days.Now, RFC6844 has this in it:
CodeSOD: I Exist
In addition to using an ancient development environment, with terrible UX, Greta also has the misfortune of working in Pascal.Recently, she was diagnosing a bug. The program was reporting that files didn't exist when they definitely existed. She traced the problem down into the system library. Let's see if you can spot what's wrong:
CodeSOD: An Odd Sort
Let's say we wanted to query Active Directory and print out a report of all of our users, and their last logon time. That seems like a pretty normal task for a Powershell script. It'd probably be short and easy to read, at least if it were written by a normal person.Alice sends us one that wasn't. She's already done us a favor, as she writes: "Code cleaned up and indented for the whitespace-missing-impaired."
CodeSOD: Extremely One Line
Autoformatting your code is a standard thing to do these days. And in those days past, if we're being honest. There's no excuse to not use some kind of autoformatter. Whether you configure your editor to do it or are a weirdo like me who runs a formatter from the CLI as a build step, you've got an easy way to format your code so it looks neat and readable. And some IDEs, like Visual Studio, are pretty insistent about doing this for you. Which makes today's code sample a bit more perplexing. This comes from an ancient ASP .Net application that Austin has the misfortune to work with:
CodeSOD: Vintage 2013
Today we have more of a representative comment, from Watson. This comes from some GPL licensed code published by everybody's favorite evil empire, Oracle.
The Weakest Leg
On July 10 2026, a life insurance company called TruStage had a "cybersecurity incident". According to an update posted five days later, when they revealed the incident, they are "continuing to work carefully and urgently to understand the facts, and the company will communicate appropriately as its understanding of the situation evolves."This work, according to their outage page (last updated on September 11th, at the time of this writing), is continuing. Every major business function is listed as "partially available". They have some text explaining that they can't simply turn the system back on, because safety.They don't know what data was compromised. They don't know when they'll have everything back up. As of mid-August they had gotten so far as setting up a "clean" environment that they could start migrating services to. The only good news is that some of their services are supplied to credit unions for banking, and those were on different systems, so only their insurance packages were compromised. I also suspect the fact that they serve credit unions is the only thing keeping the business afloat, because failing to handle transactions for months seems like a terminal event for most businesses.Obviously, this also means they're embroiled in an ever growing number of lawsuits from a variety of parties. It also highlights something about how insurance works: it's middlemen all the way down.TruStage will issue policies, but their connection to the actual policy holder will likely be routed through a number of different partners. Like, for example, Ethos.Ethos uses "AI" to "democratize" insurance. Feed your data into their AI, and in minutes they'll pair you up with a policy. Which, while using machine learning to do things like assess insurance risk seems like a pretty reasonable idea, in this age of AI hype, one has to wonder how this actually is implemented and how it works in practice.But Ethos isn't the end of the chain! Other companies live downstream from Ethos, for example, Family First Life. A customer might reach out to FFL to get a policy, FFL reaches out to Ethos (or one of many other partners), and Ethos reaches out to one of its partners (TruStage being one of their largest), and boom: an insurance happens. Like I said, it's middlemen all the way down.But now, here's the problem: TruStage issued a bunch of policies, and then stopped being able to do business. They stopped being able to do even vital things, like process payments. And you know what happens when payments aren't made on a life insurance policy? It lapses. It goes away. You no longer have insurance. And the agents who sell those policies are often paid a commission based on the value of the first few months of the policy's life (since life insurance is expected to be a long term investment). If the policy lapses, those commissions vanish, companies like Ethos (or downstream partners like FFL) get chargebacks on the value of those failed policies.This makes some people, like FFL's President, Shawn Meaike, very unhappy.What we have here is a pretty egregious failure of cybersecurity and disaster recovery on the part of TruStage. It's an embarrassing and potentially terminal event for them. It's awful for the customers who bought insurance policies to give themselves peace of mind, only to discover their policies lapse because TruStage can't take their money. It's creating mild chaos in the entire industry. That's all interesting, but not why I wanted to write an article about this. I wanted to write about the incredible cringe.In this clip from an insurance industry conference, Meaike brings up all of their partners for a "carrier panel". He then proceeds to line them up from stage right to stage left, ranking them as "weakest" to "strongest". Ethos, who sells TruStage policies, is labeled as the weakest."You have the weakest leg," he says, pointing at the reps from Ethos, to laughter from the audience, and then indicates the other side of the line represents "the strongest leg." Meaike hands the microphone to an Ethos employee, Dylan Cummings, "We'll start with you.""I want to start out by thanking everyone," Cummings says.Meaike cuts him off. "Hey, hey, hey, hey. I'm gonna help you. Why don't you start by saying you're sorry."The crowd yells out, "Yes!" and applauds this idea."Say 'We have a carrier that's shut down'," Meakie continues, "'and I'm mother-freaking sorry.'"Yes, he said "mother-freaking", on stage.So let's recap. An insurance company gets pwned so hard they can't run operations for months. People's policies lapse, simply because they can't give their money to the insurance company. Which 100% sucks for those customers, though at least with life insurance we expect very few of them are missing their payout- hopefully! This chain cascades through the network of partners, until it culminates in the president of one company publicly humiliating the senior account manager at a middleman company, during an industry conference.Nobody comes off looking particularly good here, obviously. Meakie is a jerk. TruStage is institutionally incompetent. Ethos is in a bit of a rock-and-a-hard-place situation, and Cummings is trying to do his best to tap dance his way out of oblivion, but it seems like the AI driven insurance startup just isn't a good partner even before their main issuer died.All in all, it feels like a Tim Robinson sketch, or perhaps a Nathan Fielder bit. We're one step away from Meakie calling Cummings a "Wizard of Loneliness."[Advertisement] Plan Your .NET 9 Migration with Confidence
CodeSOD: The John Cage Variable
David C sends us a true confession.
CodeSOD: Historical Pads
Tim inherited a fairly antique Visual Basic application some time back. Yes, Visual Basic, not VB .Net. The application is old enough that we might consider it "vintage" or "historical"; it certainly dates from before the millennium. But it has its own unique approach to handling historical dates:
12345678