The Glitch Who Stole Christmas

Every Who down in Whoville like Christmas a lot"
But the Bug, who lived just North of Whoville, did not!

Peter L sat on his recliner, wireless keyboard and mouse balanced on his lap, and watched Jebediah Kerman burn up on re-entry. It was "Cyber Monday", also known as "look, the consumer didn't buy enough stuff on 'Black Friday', so please buy more stuff!"

Peter was enjoying his day off in part, because he was good at his job. He worked for an ad company, running the analytics and making sure the right ads got served up to the right customers at the right times. And here, on the busiest online shopping day of the year, no one was calling him, no one was screaming that the campaign wasn't running, no one was going to get him out of this-

"Honey, Macy's isn't working," his wife called from the other side living room. "Can you take a look?"

"What do you mean it's 'not working'? Are you getting an error?"

"No. It just hangs on this loading screen when I try and check out."

"Just refresh it," Peter said as he started trying to redesign Jebediah's next Kerbal death trap.

"I did that. And I tried another browser. And it's not in private mode. And some of the gifts in the cart are yours."

Peter sighed, put his game aside, and levered himself out of the recliner. His wife walked him through the steps: she could browse the store fine, she could add items to her cart, but when she clicked the checkout button, it would go to a page that displayed nothing but a "Loading"" modal popup.

Peter noticed that this was pointing at a different domain- www1.macys.com instead of www.macys.com, which made him wonder if Macy's had underestimated the load they'd need to service. Still, it did look like a lot of assets were getting served up successfully, so maybe there was something else wrong.

He pulled up the Firefox dev tools and checked the logs. Sure enough, a huge pile of errors was dumped out, and most of them revolved around assets which had failed to load- assets which weren't hosted on Macy's servers, but one of those awkward and anonymously named ad servers.

For a moment, Peter's pulse shot up, fearing that his company might be responsible for taking Macy's down. A quick check on the domain proved that it actually belonged to a different ad company. They were attempting to load a tracking pixel and that load was failing and the breaking other scripts on the page.

Peter saved the page locally, commented out the offending tracking elements- and there were a lot of them- and then loaded up the local version of the page. A quick click of the submit button sent their orders up to Macy's servers and along with it, all the gifts for their aunts, uncles, nephews, and Peter.

And what happened then? Well" in Whoville they say,
That some anonymous ad server's capacity grew three sizes that day!

[Advertisement] Scout is the best way to monitor your critical server infrastructure. With over 90 open source plugins, robust alerting, beautiful dashboards and a 5 minute install - Scout saves youvaluable engineering time. Try the server monitoring you'll today.Your first 30 days are free on us. Learn more at Scout.