Microsoft’s Linux repositories were down for 18+ hours

Enlarge / In 2017, Tux was sad that he had a Microsoft logo on his chest. In 2021, he's mostly sad that Microsoft's repositories were down for most of a day. (credit: Jim Salter)

Yesterday, packages.microsoft.com-the repository from which Microsoft serves software installers for Linux distributions including CentOS, Debian, Fedora, OpenSUSE, and more-went down hard, and it stayed down for around 18 hours. The outage impacted users trying to install .NET Core, Microsoft Teams, Microsoft SQL Server for Linux (yes, that's a thing) and more-as well as Azure's own devops pipelines.

We first became aware of the problem Wednesday evening when we saw 404 errors in the output of apt update on an Ubuntu workstation with Microsoft Teams installed. The outage is somewhat better-documented at this .NET Core issue report on Github, with many users from all around the world sharing their experiences and theories.

The short version is that the entire repository cluster that serves all Linux packages for Microsoft was completely down-issuing a range of HTTP 404 (content not found) and 500 (Internal Server Error) messages for any URL-for roughly 18 hours. Microsoft engineer Rahul Bhandari confirmed the outage roughly five hours after it was initially reported, with a cryptic comment about the infrastructure team "running into some space issues."

Read 2 remaining paragraphs | Comments