A Mean Game of Pong
Mark J spent some time as a contractor in the Supply & Inventory division of a Big Government Agency. He helped with the coding efforts on an antiquated requisition system, although as he would come to find, his offers to help went unappreciated.
The requisition system allowed a user to enter in a part number they needed and have it ordered for them. Behind the scenes, a database lookup by part number returned all warehouses that had it stocked. Then it determined the closest warehouse to where the request came from. Finally, it would create an order to have the part shipped from Warehouse X to the user's location. It was quite simple and worked fine for what Big Government Agency needed, except when it didn't.
Every now and then, chosen Warehouse X would reject the order. Milliseconds later, it would run back through the system and be rejected again Warehouse X. As a result, the message bounced back and forth thousands of times, creating enough traffic to crash the system.
Eager to be a problem solver, Mark asked Jacob, the head project manager, about the order rejection flaw. "It's a timing issue, depending on when the order is made," Jacob explained. There's a separate copy of the database on the server and at the warehouse. They sync up every night, but if the warehouse shipped its last of a part during the day and someone else orders it, it gets rejected."
Ignoring the asinine practice of not having a centralized database, Mark pressed on, "Ok, but why does the same order then come back to the same warehouse? Shouldn't it find the next closest warehouse and create an order from there?"
"I suppose that would be ideal," Jacob said, shrugging. "But the rejection just sends it back to the routing engine unaltered. So it's treated as a brand new order with the same out-of-sync inventory count. Thus, it's sent right back to the same warehouse, and so on. It's this dumb thing we call ping-ponging."
Mark furrowed his brow at Jacob's acceptance of this fundamental logic flaw. "That sounds absurd. Say the word and I will find a way to address this ping-pong nonsense. Then if I have time, I'll re-architect the data structure to have up-to-the-minute inventory statistics kept in a single copy of the database."
"Mark, we have too much else to do. We can't be worrying about technical debt like this," Jacob responded, growing irritated. "When it causes the system to crash, we just start it back up and go about our business."
Mark decided to drop his argument and began calculating how long his Big Government Agency contract had left. The following week he came in to find a ping pong paddle on his desk. Next to it lay a note from Jacob, "Hey Mark. Since you're so interested in helping with the ping-pong problem, I'm assigning you the duty of restarting the system whenever it goes down. Thanks for being a team player. ~ Jacob."
[Advertisement] Forget logs. Next time you're struggling to replicate error, crash and performance issues in your apps - Think Raygun! Installs in minutes. Learn more.