The Secret is… … … … Timing

Scott was new to the team, and so when a seemingly simple bug came in, he picked up the ticket. It should be an easy win, and help him get more familiar with the code base.

The reported problem was that a user entered the time, and saved it. Several screens later, when that time was redisplayed, it was incorrect, off by some number of hours. Clearly, it was a small timezone issue, which should be easy to fix.

Except that was easier said than done. Scott's company made a hosted service, and quite reasonably, each customer had their own database instance. But they also all had their own branch in source control. That's where things started to get problematic, as there was no coherent naming convention, each customer's version had its own customization and a blend of merges and cherry-picked commits that made it impossible to, at a glance, understand what was or wasn't deployed to any given branch.

Once Scott figured out which branch to use, he stood up a simulation environment and tried to replicate the error. He entered the date. He checked the database- the date was stored correctly. He went to the screen which was incorrect, and saw that it was off- though by a totally different number of hours than the customer reported. On a different screen, however, the date displayed correctly. Digging through the code, the incorrect screen was adjusting the timezone from UTC to the local timezone, while the correct screens weren't doing any timezone adjustment.

Confused, Scott went to his boss, Zofia. "I assume it's doing the adjustment when it shouldn't, but I'm just confused by why it's doing that?"

Zofia sighed. "Because a very long time ago, I let the owner talk me into a very bad idea. We don't store timezone information with the times. Most of the time, we convert them to UTC and store that, and then convert to the local timezone on display. But because we were in a 'move fast and break things' phase, we didn't do that with all of our times. Some we just store in the local timezone- without storing any timezone info."

The result is that a timestamp, stored in the database, might either be in the local timezone of the person who created the record, or might be in UTC, but you couldn't tell which by looking at the database. So you had to check the code which handled the storage and display. That code was in a twisted network of git branches, so for any given customer, the rules might be utterly different, and since commits were sometimes cherry-picked across branches (as a useful feature for one customer becomes useful for another), the rules might suddenly change if you weren't careful.

"Oh... that's... horrifying."

Zofia nodded. "You think that's bad, keep in mind that our reporting engine is a third party product that doesn't use the branching structure we set up in git. One change to dates in the application could mean you spend the next few days tracking down every reference to it and making sure you're only making changes that impact the one customer and nobody else."

"I... maybe I should pass this ticket off to someone else?" Scott was certainly a lot less interested in tackling this "simple" bug.

"Nah," Zofia said. "It'll be a good way to learn the codebase. This is what we do every day."

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!