A Cable Outage

New Year's Eve is a wonderful day. Not only a time of rejoicing-meeting and partying with friends and strangers alike to celebrate making it through yet another year-but also a time of change. Between new resolutions and yesteryear's memoirs, it's the best moment to move our lives in a different, exciting, and surprising direction.

For Mitch, however, that was not the case. Working for a cable TV company had taught him that when your superior called in the early evening on your day off, it meant one thing and one thing only.

"Hi, Boss," he sighed. "How bad is it?"

"Bad." The boss didn't sound too excited himself. "HBO and Showtime are down. People will want our heads on pikes if we don't bring them back up ASAP. So, I know I shouldn't be bothering you, but I've tried everyone and-"

"No, it's okay," Mitch said. "Just give me twenty minutes to get to the office. I think I know what the problem is, it shouldn't take long."

"You're a lifesaver," the boss said with relief. "We'll talk about compensation when you're back. See you, and good luck."

Mitch left the house and started his car in surprisingly high spirits. The receivers at the office had their bad moods. All he needed to do was flick a reset switch, and the channels would be broadcasting again. Given holiday overtime was six times his normal hourly salary, he felt like he'd gotten the better end of the deal.

Two hours later found Mitch kicking himself for his optimism.

He'd tried resetting the receivers. He'd tried replacing them with spares. He'd checked and rechecked all the cables and connections, went through the whole troubleshooting list twice-all for naught. Not only were the two premium channels still not broadcasting, but along the way, another three channels had gone down, one by one.

Think, Mitch, think! he urged himself. Huddled in front of a laptop, he tried to connect to one of the failing devices over the local network, but it was just as unmoved by his pings as by his pleas. There was nothing visibly wrong with the network configuration, and other devices plugged into the same ports worked just fine, but the dead receivers stubbornly refused to work no matter what tricks he used.

Mitch had no choice but to move to the next step in the troubleshooting protocol. With trembling hands, he dialed the device vendor's helpdesk number.

"Thank you for calling Initrode Systems, my name is Nathaniel. How can I help you?"

A New Year's miracle! The monotone voice on the other side filled Mitch with hope. He'd never expected to connect to a living soul so quickly, on a holiday at that.

"Hello, my name is Mitch. I'm having a problem with my equipment ..." He explained his troubles in detail, noting every symptom he'd witnessed.

"I see," Nathaniel said. "Have you tried turning the device off and on again?"

The lengthy conversation left Mitch no wiser. He went through the motions, only to hear that "the only suggestion" Initrode Systems had for him was to replace his router. Mitch couldn't fathom how a router would have anything to do with not being able to connect to devices on the same network. Nevertheless, he grumbled a thank-you to Nathaniel and hung up.

One thing he'd noticed during the troubleshooting gauntlet was that right after turning on, the devices would work for a few seconds, and only then suddenly drop dead. That gave him a glimmer of an idea. He dug out a dusty Ethernet hub, switched up a few cables, and started capturing packets from the whole network, trying to find a pattern.

As burst upon burst of network traffic appeared on Mitch's monitor, he finally started smiling. There was a pattern to this madness. He picked up the phone and called Paul, his colleague from IT.

"Mitch! Hey there, buddy, what's up? Celebrating the New Year already?"

"No way, how could I start without you?" Mitch joked. "Listen, Paul, sorry for the interruption, but I have a bit of a situation here at work. Think you could spare a few minutes?"

"Oh. I thought you were on vacation?"

"I was."

"Well, shoot," Paul said. "I'll do my best."

Mitch recapped the outage situation, then segued into his latest findings. "A few seconds after booting, some of our receivers try to send an SNMP trap to some IP, then the ICMP Destination Unreachable message pipes in and they just die. They don't work, they don't respond to pings, they don't generate any traffic."

"Huh." Paul muttered something under his breath. "That's an odd response, but it's not wrong. Maybe there's a bug in the firmware? Have you called the vendor?"

Mitch suppressed a flare of anger. "They've been as helpful as you'd expect."

"Of course. And to think we're paying for that support ... Anyway, what was that IP?" Paul asked.

After dictating the address, Mitch heard faint typing for a few seconds. Finally, Paul picked up again.

"Well, I don't think I can help you. The IP is for a router at our other office. I can't remote in there, and you probably won't be able to get in today either."

Mitch sighed, at the end of his rope. There's one more thing I can try, he thought. "But you can remote to the router in here, right?"

"I can," Paul said.

"Can you add that IP as its secondary interface? Maybe it will play nicer with the trap?"

"That ..." Paul paused for a while. "Hey, that might actually work. Let me try."

After a few minutes, Paul told Mitch to try resetting the receivers again. He flicked the switch and went back to his laptop, crossing his fingers.

There's the trap ... there's the response ...

"Bingo!" he exclaimed as the packets started to flow again and the receiver showed up as operational.

"Woo-hoo, we saved the New Year!" Paul cried, jubilant.

"It seems so, everything looks OK here. Thanks a lot, man."

"Not a problem. Now hurry up and get out of there, you can still catch the ball drop on TV!"

"If I don't get drunk before then. See you next year!"

Mitch hung up and started packing his things. Looking through the window at the first fireworks set off by impatient celebrators, he knew what his first New Year's resolution would be: Never underestimate any problem.

[Advertisement] BuildMaster is more than just an automation tool: it brings together the people, process, and practices that allow teams to deliver software rapidly, reliably, and responsibly. And it's incredibly easy to get started; download now and use the built-in tutorials and wizards to get your builds and/or deploys automated!