OVH reveals semi-secret plan to fix critical Januscape hypervisor bug with mass reboots – and an Australian crash-test dummy

French cloud operator OVH has revealed it used its Sydney, Australia, datacenter as the crash test dummy to test a rapid rollout of a fix for the critical Januscape guest-host escape bug in the Linux kernel-based virtual machine (KVM). Januscape, aka CVE-2026-53359, allowed attackers with root access to a guest VM to execute code as root on the host, crash that machine, or take over all other guest VMs. A widespread guest-host escape exploitation is a nightmare scenario, because many major clouds use KVM to slice their servers into virtual machines and then rent those guests to clients. The prospect of attackers accessing one tenant's VM and using it to crash other guests or an entire host is therefore a terrifying violation of cloud operators' promise to run customer workloads in splendid isolation. Fixing it was therefore a priority. On Monday, French cloud OVH's CISO Julien Levrard revealed how the company handled the emergency patch job on tens of thousands of hosts that run approximately a million virtual machines, in a lengthy post that offers an unusually detailed and candid account of how clouds cope with major security incidents. One way to mitigate the bug was to disable nested virtualization by creating a two-line config file on each host running Linux KVM. That wasn't an option because OVH has no way to see if its tenants need nested virtualization and relies on it to shift VMs to different physical hosts. Nor was applying a live patch palatable, as it could introduce instability. Live migration to patched hosts was another option, but one OVH rejected as the process is slow and the company felt it might take months to move its entire fleet of tenant VMs to a Januscape-free environment. The company therefore decided to backport a Januscape fix into the Debian distribution it uses in production and reboot all hosts, giving customers advance notice but no choice. That meant customers who rely on a single host would experience some downtime, but OVH's executive committee signed off on this approach for three reasons: The need to patch before attacks; Treating cases individually would mean the OVH cloud would be vulnerable for longer; A desire to protect the greatest number of customers, and tolerate impact on a minority. OVH also decided to stay quiet about the patching plan. Communicating in more detail during the execution of the mitigation plan, while the infrastructure remained unpatched, would have significantly increased the risk for our customers, potentially leading some to test' the publicly available exploit," Levrard wrote. Danger down under To test its patching prowess approach, OVH decided to first fix its Sydney region - one of the company's smaller regions and one which, thanks to Australia's east coast being eight hours ahead of France, teams in Europe do the job during their business hours - but at a slow time in Sydney. The company therefore planned to go down under and learn from the experience before deploying the fix more widely. OVH's plan called for reboots to occur in waves, with a shutdown threshold" imposed if 15 hosts failed simultaneously in high-density regions, or five boxes in other regions. But those waves weren't as simple as just going rack by rack. The main risk for our clients during such an operation is not the reboot itself, but the simultaneous interruption of multiple instances of the same project, ensuring application resilience capable of handling a provider failure," OVH's blog on the project explains. A client who has distributed their workloads across multiple hosts to ensure high availability should not see all of their instances fail over at the same time. The decision was made to go beyond simply adhering to the anti-affinity rules that may have been defined in client deployments." Thus, for each client project with instances distributed across multiple hosts, our orchestrators calculate a co-location graph. At no time are two hosts running instances of the same project rebooted in the same window: mutually exclusive waves are defined, and a host must be back online before the next one is launched in the same anti-affinity class." The patching effort hit a few snags. Some VMs didn't restart after hypervisor reboot. Some experienced data corruption during forced shutdowns. OpenStack APIs misbehaved, producing hours of HTTP 503 errors and necessitating postponement of one patching wave. At a Canadian site, API traffic reached 10 times the usual peak, overwhelming the manager and support teams." Hardware hiccups Some hardware died during the process. On the first night, approximately 20 to 30 hosts out of 6,000 did not recover on their own," Levrard revealed, blaming faulty memory modules, BIOS configuration issues, inactive network interfaces." Some machines needed new CMOS batteries. The CISO thinks OVH's approach was a remarkable feat" as it produced a very reasonable number of outages and customer impact relative to the scale of the project." But he thinks the company will need to improve. Recognizing that the coming months could see further kernel vulnerability disclosures, it seems clear that this emergency procedure will need to be repeated," he wrote. We will need to do better next time, both in managing the raw impact of the restarts and in providing customers with advance notice and support during the operations." The company is therefore conducting a post-mortem analysis it hopes will improve its processes. (R)