Your Cloud Survived Everything Except The Real World
Arthur T Knackerbracket writes:
You don't know what you've got till it's gone. Great lyric, lousy data retention policy.
Amazon Web Services said last week that war damage to its Middle East infrastructure had permanently destroyed resources and data hosted exclusively in its now rather badly named Bahrain Availability Zones. The damage overwhelmed the resilience built into the region. Customers without copies elsewhere no longer had their data. Sorry about that.
This may have surprised anyone who mistook cloud redundancy for an intrinsic guarantee of safety. AWS is far from the only American operation to have suffered in the region: the US Navy has reportedly had its local maintenance and supply network badly mauled, with serious consequences for its operations.
If systems designed to withstand war cannot cope with sustained physical attacks, civilian bit barns have little chance. The episode also underlines a familiar but easily neglected lesson: resilience within one cloud region is not the same thing as maintaining an independent backup elsewhere.
Physical destruction is not the only threat. A major outage of the UK air traffic control system in September, which stranded hundreds of thousands of passengers and led to thousands of flight cancellations, was reportedly triggered by a military aircraft filing an incompatible flight plan. Presumably Flight Lieutenant Bobby Tables has been reprimanded.
The apparent failure to validate the flight plan data was not the worst of it. NATS, which runs the UK's air traffic control system, reportedly told airports and airlines that no backup system was available because it was undergoing a "complete overhaul." Nor was there a backup of the live data, supposedly because of the "vast amounts" involved.
If your system produces too much data to back up, you had better be running a particle collider or a giant telescope. Otherwise, you may be in the wrong business. More charitably, backup strategies are complicated, expensive, and difficult to test. They also suffer from the insurance problem: while nothing is going wrong, management sees only capital and operating expenditure with no obvious return.
The AI infrastructure boom has made that problem considerably worse. A backup is a copy, and a copy needs storage. AI datacenter operators are swallowing much of the available capacity, with drives selling out faster than tickets for a Taylor Swift tour. Western Digital had allocated its entire 2026 hard-drive production run by mid-February. The only consolation for those responsible for keeping data safe is that they can say "Yeah? You buy it, then" to anyone who smugly invokes the 3-2-1 backup rule. Three copies on two different media with one kept off-prem? Lovely idea, if you don't have to provision it. Someone is provisioning it for all those giant datacenters that will run our lives, right? Right?
Read more of this story at SoylentNews.