Sharked

Andrew M. worked at a small company in Kansas City called EtherTrode. With one facility and about 20 employees, they designed and built custom Ethernet hardware and drivers to fill niche roles where the common integrated chipsets weren't good enough. Their hardware worked quite well, which attracted the attention of a multinational conglomerate called Initech. Initech puchased EtherTrode, rather than develop their own Ethernet devices.

Like most others at EtherTrode, Andrew was a jack-of-all-trades. On any given day, he might be doing anything from customer support, to application development, to general office IT tasks, and even once a bit of soldering. It was the general office IT tasks that got Initech's attention: he was selected to work with Initech on integrating their networks.

The first phase went more-or-less smoothly. Initech shipped Andrew a pre-configured Cisco router, and instructions on how to connect it. After a weekend of integration, EtherTrode's network now had a permanent VPN tunnel to Initech's headquarters in Detroit.

Phase One was boring. Yes, there were minor screwups and a few WTFs due to some miscommunication and unnecessary outsourcing, but that's a variation on the same story that's been told a million times before. Phase Two, however, was the Grand Slam over the foul line, lightning in the whiskey warehouse, the Cheese Shop without cheese, and a painful urination upon a subway's third rail"

" Active Directory Integration.

Once the networks were connected, Initech wanted to terminate EtherTrode's Active Directory domain and move all of their accounts and systems into Initech's domain. Contractors working for Initech prepared a new domain server and shipped it to Andrew. He received the system, racked it the next weekend, and powered it on. Everything seemed to work, and local systems joined to Initech's domain without issues. Come Monday morning, the office was ready to resume business.

Things turned sour around lunchtime on Monday. All morning, employees had complained about poor network performance. Their emails failed to send, and new messages failed to arrive. Tiny Subversion check-ins would fail more often than not, and even internal web pages took minutes to load- if they loaded at all. Worse still, Initech's security officers had installed a new electronic lock system on EtherTrode's building, which was connected to the LAN. By lunchtime, the locks refused to unlock, and Andrew had to physically unplug the controller unit in the server room to disable the lock so employees could actually enter and exit.

The core staff of EtherTrode had an impromptu meeting shortly after lunch to discuss the issues. Lyle, another engineer, arrived late. "But I know what the problem is," he explained.

"I noticed the lights on our core switch were blinking like a strobe light in a tornado, so I fired up Wireshark." A few of the engineers nodded, and Andrew wondered why he hadn't thought of that. EtherTrode made Ethernet equipment, and Wireshark was an invaluable tool for examining the behavior of their devices and networks. It was the best way to make sure their hardware complied with Ethernet and TCP/IP standards, and it was really good at not interfering with their network while doing its monitoring. "In short," Lyle explained, "there's a device at 192.168.16.245 that's issuing tens of thousands of broadcasts per second. The switches are dropping frames, and the IP stack on everyone's workstation is working overtime processing irrelevant packets."

"254, you say?" Andrew said. He frowned. "That's the IP address of our new domain server""

"Can we shut it down?" asked Ryan, their boss. "Rollback to how our network was last week?"

Andrew shook his head. "I've already joined everything to the new domain. If we shut down the controller, no one will be able to log in or access network resources" not that they can now."

Lyle spoke up again. "The traffic looks pretty fishy. I think the server's got some sort of malware on it. Can we log in and see what it's doing? Run an antivirus or kill the process?"

Andrew shook his head again. "Initech has it locked down tight. I can do some basic domain admin tasks, but I can't actually log into the server."

"Alright," Ryan said, visibly irritated, "Lyle, give me your Wireshark capture. I'm going to go down to the coffee shop for their wi-fi, and I'm going to forward it to Initech and raise some hell."

In the meantime, Andrew and Lyle went back to the switch. Since the broadcast was on a single port, they blocked that port at the switch. The server kept shouting its head off, but now the switch dropped the packets. Doing anything that required talking to the domain server- like logging in- was painfully slow and failed half the time, but the rest of the network worked fine. That bought them some time while they waited for a better solution from Initech.

The next day, Andrew came in, grabbed a mug of coffee, and picked up the top priority support ticket. This one was a customer support issue. They had uploaded their own Wireshark capture to an FTP server. Andrew downloaded it so that he could see why their EtherTrode equipment was misbehaving. When he went to open the capture, Wireshark refused to start- or more accurately, it was no longer installed on his system.

"That's strange," Andrew thought to himself. Maybe a Windows Update messed up the registry settings, or something, so Andrew redownloaded Wireshark and ran the installer.

Or tried to. The installer refused to run. "This publisher has been blocked by your administrator," explained the error message. As if on cue, his email client dinged and he was a new memo from the Initech IT department:

TO: All Initech Employees SUBJECT: Re: Network Breach at KC Facility

Our new Kansas City facility had an unexpected network attack yesterday. We've analyzed the data and determined that a freeware application called Wireshark is to blame. This application caused severe issues with the facility's security system and cannot be trusted. To prevent future attacks, we have globally blocked Wireshark via Group Policy and our antivirus suite. Wireshark is just malware and this decision will not affect day-to-day operations at any Initech facility.

Andrew dropped his forehead right into his desk's surface, and all around the office he heard a chorus of thumps as his fellow engineers did the same. Initech refused to flex on banning Wireshark, and refused to admit that anything could be wrong with their domain controller.

[Advertisement] Scout is the best way to monitor your critical server infrastructure. With over 90 open source plugins, robust alerting, beautiful dashboards and a 5 minute install - Scout saves youvaluable engineering time. Try the server monitoring you'll today.Your first 30 days are free on us. Learn more at Scout.