CodeSOD: Scheduling Buttumptions

Steph had been at this job long enough to be fairly good at it, but not quite long enough to have peeked in all the dark corners yet. As such, when she heard that there was an issue with scheduled jobs, her first thought was to poke through cron to see if she could pick out what schedule was misbehaving. Apparently, all of them- cron was empty.

Confused, she went to her team lead Greg, asking about where she might find the scheduling setup. And that was when she heard about Travie the Whiz Kid. A junior developer with no degree, he'd been hired solely based on his ability to talk a big game about how he single-handedly saved several companies by providing them with innovative websites during the dot-com bubble... when he was twelve. The Whiz Kid was a Special Snowflake; he preferred to reinvent the wheel rather than implement stable but "boring" code. Upper management was convinced he was an unparalleled genius, and had exempted him from the usual QA standards. Unfortunately, he'd grown utterly bored with Business Intelligence and transferred to the Web team, leaving his inventions behind for Steph to maintain.

Travie had been tasked with writing a helper application in Java to interact with and process data from a third-party web service. The third party processed their data on certain days at certain, known times, and their app needed to wait and pull down the data after it was guaranteed to be there. The ticket involved the wrong data being collected: at 5:15AM on the second of the month, the script was pulling in the previous month's data to process, rather than the current. Was it running early?

Steph found the repo with the Java code The Whiz Kid had written and checked it out, skimming over the list of files as she finished her morning latte. She opened the Schedule class, winced, and closed it again. "Definitely need more coffee."

 private static int eighth_hour_of_day = cal.getActualMinimum(Calendar.HOUR_OF_DAY) + 8; private static int ninth_hour_of_day = cal.getActualMinimum(Calendar.HOUR_OF_DAY) + 9; private static int eleventh_hour_of_day = cal.getActualMinimum(Calendar.HOUR_OF_DAY) + 11; private static int zero_minute_of_hour = cal.getActualMinimum(Calendar.MINUTE); private static int first_day_of_month = cal.getActualMinimum(Calendar.DAY_OF_MONTH); private static int fourth_hour_of_day = cal.getActualMinimum(Calendar.HOUR_OF_DAY) +4; private static int fifth_hour_of_day = cal.getActualMinimum(Calendar.HOUR_OF_DAY) +4; private static int sixth_hour = cal.getActualMinimum(Calendar.HOUR_OF_DAY) + 6; private static int last_minute_of_hour = cal.getActualMaximum(Calendar.MINUTE) ; private static int second_of_month = cal.getActualMinimum(Calendar.DAY_OF_MONTH) + 1; private static int fifteenth_minute = cal.getActualMinimum(Calendar.MINUTE) + 15; private static int last_day_of_month = cal.getActualMaximum(Calendar.DAY_OF_MONTH); private static int last_hour_of_day = cal.getActualMaximum(Calendar.HOUR_OF_DAY)-2; private static int thirty_fifth_minute = cal.getActualMinimum(Calendar.MINUTE) + 35;

A second, larger latte obtained, Steph took a deep breath and opened the class again. By the time her third latte was finished, she'd tracked down the issue: fifth_hour_of_day was accidentally, ridiculously defined as 4, meaning the code was pulling down the data at 4:15AM, before the new month's data was available.

Code fixed and checked in, Steph rapidly deleted her working copy and closed the ticket. At least I'll never have to-

"Good work, Steph!" Greg said. "I'll send you the other ticket now."

Head met desk.

The second ticket turned out to be no more obvious than the first at first glance. Now that the constants were fixed, there seemed no reason for a second event, scheduled for 4:00AM on the dot, not to fire. This would require debugging the while loop that did the scheduling, which she had avoided as much as possible during the first ticket:

while (true) { try { Thread.sleep(60 * 1000); Properties sysProperties = new Properties(); // Re-read properties file FileInputStream f = new FileInputStream(System.getProperty("jboss.server.home.dir")+"/conf/xxxxxxxxxx.properties"); sysProperties.load(f); f.close(); //Call Web service every day 8am Calendar today = Calendar.getInstance(); if ( today.get(Calendar.HOUR_OF_DAY) == eighth_hour_of_day && today.get(Calendar.MINUTE) == zero_minute_of_hour) { PendingData.getPendingData(sysProperties); } //Call Web service every 1st of the month at 4am if (today.get(Calendar.HOUR_OF_DAY) == fourth_hour_of_day && today.get(Calendar.DAY_OF_MONTH) == first_day_of_month && today.get(Calendar.MINUTE) == zero_minute_of_hour) { AllData.updataDatabase(sysProperties); } //Call Web service 2nd of month at 5:15am if (today.get(Calendar.HOUR_OF_DAY) == fifth_hour_of_day && today.get(Calendar.DAY_OF_MONTH) == second_day_of_month && today.get(Calendar.MINUTE) == fifteenth_minute){ OtherData.getOtherData(sysProperties); //Anon'd } // etc etc etc } catch ( /* etc */ ) { }}

Of course, the solution is likely obvious to you by now: what happens if it takes longer than a minute to update the database? One of the snipped instances, far down the page and added later, executed at 03:59:59.987, taking just over a minute to complete, preventing the 4:00AM job from executing. They both had to execute, but obviously were incompatible with each other given the current architecture.

Finally, Steph went to her team lead. "I can't fix this," she said, rubbing her temples. "I guess I'm just not enough of a whiz to figure it out."

Greg made a face. "Don't tell management, but I've got a prototype using an actual scheduler I suggested when this was made."

"I'll replace the whole damn thing from scratch on one condition," she said, holding up her empty Starbucks cup.

"Don't worry, I've got this," Greg said.

Later that day, he tracked down Travie, and encouraged the Whiz Kid to chip in on a $50 Starbucks gift card for Steph.

[Advertisement] Release!is a light card game about software and the people who make it. Play with 2-5 people, or up to 10 with two copies - only $9.95 shipped!