That 70's Paper Mill

The late Seventies was a lucrative time for Finnish-based Kirkniemi Paperi, a paper production powerhouse. Puoval had a great opportunity to cash in on the profits by helping to integrate a completely automated, computer-based production system. His degree in electronic engineering was finally going to pay off.

Thanks to the invention of the Intel 8085 microprocessor, it became possible to turn trees in to paper quicker than ever. Puoval had a mandate from Kirkniemi ownership to spare no expense with getting the system up and running since their biggest competitor implemented a similar system the year before. But if they had a bumpy rollout, it would be incredibly damaging to both the company and Puoval's livelihood.

The system would have to control paper machines more than a hundred meters long, capable of moving paper through them at 80 kiliometers per hour (which would make for one wicked paper cut). While they were literally well-oiled machines, a small problem with any part of the production would bring things to a halt due to everything being interconnected. When paper-making stops, moneymaking stops and the bosses get angry.

Despite the risk, Puoval was up to the task. He drooled over the hardware he was allowed to bring in to be the brains of the operation. He got to set up a whole army of blazing-fast 6 Mhz Intel 8085's, cages full of in-house built Eurocard logic boards, analog and digital sensors, kilobytes of tight machine code, the whole works. As a fail-safe, Puoval even set up a battery backup system, and line conditioning to prevent a power surge from frying his masterpiece.

It was a long and difficult setup process, but Puoval's brainchild sparked to life and, amazingly, worked as intended right away. The output of the paper mill increased 40% and Puoval's bank account increased 100%. For over a year, it hummed along flawlessly cranking out gloriously large rolls of paper. But then, this feel-good story got beat to a pulp.

Late one night around 11 PM, Puoval's phone rang. On the other end was the panicked 2nd shift manager of the mill. "Puoval! Are you still awake? GOOD! We need you in here ASAP! THE SYSTEM IS DOWN!"

"What do you mean it's down?" Puoval asked back through growing sleepiness.

"Well, the machines were running just fine, then out of nowhere, they just shut off and ground to a halt. Paper went flying off the rolls all over the place. It's like a gang of mummies exploded in here!"

"Ok, I'll get dressed and be in as soon as I can," Puoval sighed, unsure what to expect upon his arrival.

While the 2nd shift line technicians cleaned up the paper disaster, Puoval headed for his control system. He half expected to see it charred and smoking, but everything looked fine. He ran some system diagnostics and nothing looked out of the ordinary other than the sudden cutoff in the logs when everything stopped. He began the lengthy startup process of the system and machines and the mill was back in action.

When the bosses caught wind of the unexplained failure in the morning, Puoval was put on notice that it could not happen again. "Understood. I'm going to give everything a thorough inspection today, and stay late to make sure nothing weird is happening during 2nd shift," he assured the powers-that-be.

Midnight rolled around and everything was still humming along. Puoval decided to head home but made it clear to the manager to call him immediately if even the slightest thing went wrong. Fortunately nothing went wrong that night. Or the next night. Puoval was ready to chalk it up to a one-time freak occurrence. But the 3rd night his phone rang around 11 PM again. "PUOVAL! WE NEED YOU NOW!"

Puoval came in to find the same situation as three nights before. He got everything running again but in the morning the bosses demanded he work both 1st and 2nd shift every day until he had an answer. Two late nights came to pass without a hitch. But when the dreaded 3rd night arrived, he was on high alert. The clock ticked towards 11 PM and he began to sweat. Nothing could go wrong though, as he was watching his beloved system like a hawk.

Just as he was sure nothing would go wrong this day, an innocent-looking cleaning lady strolled up pushing a large commercial vacuum cleaner. He watched her ignore the multiple signs around the cage of his computers that said "DO NOT USE THESE OUTLETS, EVER!" and begin to plug in her vacuum. Puoval sprang at her, shouting to stop.

Not expecting his rapid advance, she had a look of dread come across her face. "But I need to clean?"

"Absolutely not! Not right here! You will bring this whole plant to its knees!" Puoval warned. Upon further explanation, the cleaning lady's behemoth vacuum was a new addition to the fleet. The smaller, economical one didn't draw enough juice to take Puoval's system down. Once this beast was plugged in to the forbidden outlet, which happened to be past the line conditioners, everything came crashing down. At least Puoval found his explanation, albeit a ridiculous one, so that he may retire from 2nd shift.

[Advertisement] Manage IT infrastructure as code across all environments with Puppet. Puppet Enterprise now offers more control and insight, with role-based access control, activity logging and all-new Puppet Apps. Start your free trial today!