Google engineer unplugged every fiber they could see and – surprise! – took down a chunk of the G-Cloud

Google Cloud has revealed the embarrassing reason for a partial outage at its us-central1-b region this week: An engineer unplugged it. The incident spanned 07:41 to 11:52 PT on September 1, and saw a portion of the G-Cloud region experience severe network degradation and resource isolation." Traffic flow drop rates for resources hosted in the affected area reached 100% during the peak of the incident, resulting in unreachable virtual machines and elevated packet loss," states Google's service health report. The preliminary root cause on that page explains that Google designs its cloudy datacenters so they have redundancy across multiple routing devices." The system is designed to be resilient to all single device or fiber-path failures, and most double-or triple-failures do not affect customer traffic. To ensure this, the devices and fiber paths are physically separated in each datacenter, with diverse power sources," the document explains. But it's possible to defeat all that redundancy by pulling out all the cables. The immediate technical trigger for this event was the inadvertent physical disconnection of network fiber-optic cables during a routine hardware maintenance procedure," Google explained. A procedural error meant that the physical maintenance action sequentially unplugged 100% of fiber paths across all devices within 13 minutes. The nature of the error, combined with the speed of the action, prevented warnings of incorrect action reaching the engineer before complete disconnection." Once the engineer disconnected the routers, VMs in one zone of the us-central1-b region lost contact with the outside world. They could at least chat with each other, but users couldn't contact their VMs to figure out what was going on. Once Google noticed the situation, it moved traffic away from the routers that the engineer disconnected. At that point its resilience provisions kicked in as traffic automatically shifted what the company described as healthy capacity elsewhere in the region." While that was going on, technicians identified the disconnected optical links and physically reseated the fibers. Once the physical links were fully restored, traffic flow rates normalized and the traffic was redirected back to return the capacity to service." To summarize: A Google Cloud zone went offline because one of the company's engineers pulled out all the cables that connected it to the internet. Once someone plugged everything in again, it started working. The Register understands that outfits like Google create detailed procedures for how to safely maintain their infrastructure, and that technicians should always follow them carefully. It therefore seems like the engineer who pulled the cables failed to observe one of the most basic rules of tech: RTFM. (R)