Google Cloud Explains How It Accidentally Deleted a Customer Account [REPRISE]
Freeman writes:
Previously on SoylentNews: "Unprecedented" Google Cloud Event Wipes Out Customer Account and its Backups - 20240521
Earlier this month, Google Cloud experienced one of its biggest blunders ever, when UniSuper, a $135 billion Australian pension fund, had its Google Cloud account wiped out due to some kind of mistake on Google's end. At the time, UniSuper indicated it had lost everything it had stored with Google, even its backups, and that caused two weeks of downtime for its 647,000 members. There were joint statements from the Google Cloud CEO and UniSuper CEO on the matter, a lot of apologies, and presumably a lot of worried customers who wondered if their retirement fund had disappeared.
[...] Two weeks later, Google Cloud's internal review of the problem is finished, and the company has a blog post up detailing what happened.
Google has a "TL;DR" at the top of the post, and it sounds like a Google employee got an input wrong.
During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer's GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period. The incident trigger and the downstream system behavior have both been corrected to ensure that this cannot happen again.
[...] In its post-mortem, Google now says, "Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and, along with third party backup software, were instrumental in aiding the rapid restoration." It's hard to square these two statements, especially with the two-week recovery period. The goal of a backup is to be quickly restored; so either UniSuper's backups didn't get deleted and weren't effective, leading to two weeks of downtime, or they would have been effective had they not been partially or completely wiped out.
[...] Google says Cloud still has "safeguards in place with a combination of soft delete, advance notification, and human-in-the-loop, as appropriate," and it confirmed these safeguards all still work.
Read more of this story at SoylentNews.