Powerful Trouble
Many years ago, Chris B. worked for a company that made CompactPCI circuit boards. When the spec for hot-swappable boards (i.e., boards that could be added and removed without powering down the system) came out, the company began to make some. Chris became the designated software hot-swap expert.
The company bought several expensive racks with redundant everything for testing purposes. In particular, the racks had three power supply units even though they only required two to run. The idea was that if one power supply were to fail, it could be replaced while the system was still up and running. The company also recommended those same racks to its customers.
As part of a much-lauded business deal, the company's biggest-spending customer set up a lab with many of these racks. A short while later, though, they reported a serious problem: whenever they inserted a board with the power on, it wouldn't come up properly. However, the same board would initialize without issue if it were in place when the system was first started.
Several weeks slipped by as Chris struggled to troubleshoot remotely and/or reproduce the problem locally, all to no avail. The customer, Sales, and upper management all chipped in to provide additional pressure. The deal was in jeopardy. Ben, the customer's panicked Sales representative, finally suggested a week-long business trip in hopes of seeing the problem in situ and saving his commission the company's reputation. And that was how Chris found himself on an airplane with Ben, flying out to the customer site.
Bright and early Monday morning, Chris and Ben arrived at the customer's fancy lab. They met up with their designated contact-an engineer-and asked him to demonstrate the problem.
The engineer powered up an almost empty rack, then inserted a test board. Sure enough, it didn't initialize.
Chris spent a moment looking at the system. What could possibly be different here compared to our setup back home? he asked himself. Then, he spotted something that no one on the customer side had ever mentioned to him previously.
"I see you only have one of the three power supplies for the chassis in place." He pointed to the component in question. "Why is that?"
"Well, they're really loud," the engineer replied.
Chris bit back an unkind word. "Could you humor me and try again with two power supplies in place?"
The engineer hooked up a second power supply obligingly, then repeated the test. This time, the board mounted properly.
"Aha!" Ben looked to Chris with a huge grin on his face.
"So, what was the issue?" the engineer asked.
"I'm not a hardware expert," Chris prefaced, "but as I understand it, the board draws the most power whenever it's first inserted. Your single power supply wasn't sufficient, but with two in there, the board can get what it needs."
It was almost as if the rack had been designed with this power requirement in mind-but Chris kept the sarcasm bottled. He was so happy and relieved to have finally solved the puzzle that he had no room in his mind for ill will.
"You're a miracle-worker!" Ben declared. "This is fantastic!"
In the end, functionality won out over ambiance; the fix proved successful on the customers' other racks as well. Ben was so pleased, he treated Chris to a fancy dinner that evening. The pair spent the rest of the week hanging around the customer's lab, hoping to be of some use before their flight home.
[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!