NTP server that traveled back in time caused massive Aussie mobile outage

Australian telco Telstra has revealed the cause of the recent incident that caused widespread connectivity problems across its mobile networks, inculding outages to Australia's 000 emergency services line, plus outages to electronic payments services and transport networks. The carrier explained itself in a submission [PDF] to a Senate inquiry into outages affecting Australia's emergency services which initially investigated an outage at Telstra's main rival, Optus. The Optus incident is linked to multiple deaths after people could not reach emergency services. Telstra's submission reveals that the carrier's attempt to address a known resilience problem was the instigating incident. That problem was a faulty backup power feed in the chassis used to house a network time protocol (NTP) server. A few minutes before midnight on July 7, a Telstra techie started work to replace that chassis. By 3:38 AM on July 8, the worker had finished the job and powered up the server. The NTP server included a GPS card that Telstra says did not operate as expected" when the server came back online. We now believe this occurred because of an intentional design change that had previously been made to the equipment to fix an earlier fault [which] had not been properly documented," the submission states. This meant the maintenance team was not aware of the way the device would behave when restarted." Telstra also admitted it had not applied a software update to the device, despite knowing of its availability in early 2026. Had that software update been completed or had the design change been properly reviewed and documented post the earlier incident, and reflected in the maintenance procedure, the outage may not have occurred," the submission states. The outage did happen because once the NTP server came online it reset its clocks to the year 2006. The server then did what NTP servers do: publish that time to myriad other machines across the network. Once those machines received the incorrect time, other kit on the Telstra network compared digital certificates and decided something dodgy was going on - so denied connections. Just one of Telstra's three NTP servers had this problem, but enough bad timing info made it onto the carrier's network to cause chaos. The telco isolated the bad box at 7:11 AM, and by 10:30 AM had identified all network components that used the bad time information the naughty NTP box broadcast. That wasn't the end of the matter, because as valid time information rippled across the network, some equipment didn't close IP sessions. Customer devices therefore couldn't reconnect to the network unless rebooted. The carrier has described the outage as clearly unacceptable." If maintenance work can trigger this kind of outage, it suggests our controls were not good enough. We are accountable for that, and our investigation will address why that design change was not documented, why the software update was not completed, and what needs to change in our controls so known risks are captured, prioritised and closed before they can affect customers." The carrier has done the usual thing by promising to conduct deep and lengthy self-reflection, submit itself to further flagellation at more inquiries, co-operate with a probe by the relevant regulator, and to do what it takes to prevent similar incidents. (R)