On-call techie decided job was done and hit the bottle – just before his pager went off

ON CALL Welcome to another installment of On Call, The Register's weekly reader-contributed column that celebrates the IT professionals who put their lives on pause to provide tech support at all hours. This week, meet a reader we'll Regomize as "Jemaine." In the early 1990s, he found himself in Hong Kong working as a database specialist on VAX/VMS systems. "We'd built a billing application for a telco client in Macau, and it had been running happily for some time," he told On Call. By the time the system needed its first major OS upgrade, Jemaine was therefore happy for the local crew to handle the job. His client had other ideas and, despite also arranging for two DBAs to be present during the upgrade, insisted he show up. This was not a hardship because the job coincided with the Macau Grand Prix and Jemaine wasn't required to be on site. The client had therefore provided him with a hotel room that, as luck would have it, had a view of the track! "A couple of friends ended up crashing my room, and we spent the weekend watching insane drivers hurl cars around an absurdly tight street circuit," Jemaine admitted. The client never called or paged, so after the race Jemaine was confident the upgrade was going well. He and his friends therefore consumed "several bottles of rich Portuguese red wine" and ordered a sumptuous meal. "Dessert had just arrived when my pager went off," he told On Call. Jemaine poured himself into a cab to his client's office and found a situation he described as "vague but clearly serious" because the billing application wouldn't start. "Judging by the silence and the stoic expressions, everyone was quietly panicking," Jemaine wrote. He soon learned that the client had already tried to fix the app by reinstalling the OS twice and had now decided the database was the source of the problem. Jemaine was told to wait while the DBAs reinstalled the database, which "gave me time to sit in a back room and sober up slightly," he admitted to On Call. The database rebuild finished at about 2 am, but the application still refused to start. The client then turned to Jemaine. "I was summoned and interrogated by the systems team," he said, and ran a quick check that showed the database was perfectly healthy - but the batch scheduler wasn't running. To probe that problem, Jemaine asked to speak with the lead developer - who, it turned out, was not on site. "An urgent page was sent, and fortunately he called back quickly. His suggestion was to step through the code. This meant compiling a large COBOL program I'd never seen before in DEBUG mode, then single-stepping through it over the phone with the developer." By now, an increasingly anxious semicircle of client staff was watching Jemaine's every move, and he felt like they were silently shifting blame in his direction. "At around 4 am, we found the failure point: batch queue submission. The call was returning a null error code. The developer was baffled." "I reached for the physical manual to see what the function actually did," Jemaine wrote. "And then, for reasons I still credit to the Portuguese wine gods, I asked a simple question: 'What account did you test this under?'" The developer immediately replied: "Administrator." Jemaine asked the OS upgrade team to run the application with administrator privileges, and it immediately worked. "The OS upgrade had introduced a new permission requirement for submitting jobs to the batch queue," Jemaine told On Call. So this was very much not his problem, and he was able to excuse himself and stagger home as the Sun started to rise. "Nobody from the company ever mentioned the incident to me again," he told On Call. "And I can't remember the name of the wine we were drinking." Have you been on call, decided nothing could possibly go wrong, and then been caught out? If so, click here to send On Call an email so we can tell your story on a future Friday. (R)