Singapore cluster outage — 5 October 2026

By Onidel Engineering team

On 5 October a failing memory module in one of our Singapore hypervisors, sg07, stalled cluster coordination badly enough that all 17 hypervisors rebooted themselves within 40 seconds of each other. Every service in Singapore was down until 04:30, when we started bringing VMs back in batches. The last ones were running again at 10:30, so depending on when your VM was restored, it was down for between two and eight hours. There was no data loss.

All times are Singapore time (SGT, UTC+8).

1. Summary

At 02:18 on 5 October 2026 we lost the whole Singapore cluster. Customer VMs, Block Storage and Object Storage were all unavailable. We started bringing VMs back in controlled batches at 04:30, and the last of them were running again at 10:30.

It began with a hardware fault on one server, sg07. Its memory started throwing errors. The hardware corrected each one, but doing so froze the server for a fraction of a second every time, and towards the end the errors were arriving close to once a second. Our hypervisors are in constant contact with each other, and while sg07 kept freezing, that conversation ground to a halt. Each hypervisor is built to restart itself if it can't confirm what the rest of the cluster is doing for 60 seconds. All 17 hit that limit within 40 seconds of each other.

The failure itself was over in a few minutes. Most of the eight hours went on getting things back safely. Three hypervisors wouldn't boot until datacenter staff stepped in, the storage cluster had to rebuild its redundancy, and the sudden shutdown left disk locks behind that we had to clear. We also brought around 1,800 VMs back in small batches, because starting them all at once could have overloaded storage and taken us down a second time.

sg07's VMs are now running on an AMD EPYC Milan server. A replacement part is on the way, and we'll move them back once it's fitted.

2. Detection and response

On-call was paged at 02:16, two minutes before the outage. Looking back, the logs had been showing trouble since 01:49, but we weren't alerting on the corrected-error rate or on the early storage slowdown, so nothing fired until it was nearly too late.

3. How the cluster coordinates

In Singapore the same 17 nodes both run customer VMs and store their data in Ceph. To follow what happened you need to know three things about how those nodes stay in agreement, and mostly how those three things interact.

The talking stick (corosync totem token)

The hypervisors have to agree all the time on things like which node runs which VM and which disks are in use. Instead of everyone talking at once, they pass a single token around a ring, many times a second. A node can only send cluster messages while it holds the token, and when it's done it hands it on.

So if the node holding the token freezes, everyone else is stuck too. The 16 healthy nodes can't send anything until the token turns up again.

Removing a stuck member (token and consensus timeouts)

The cluster doesn't wait forever. Each node keeps track of how long it's been since it last saw the token, and after 12.75 seconds (the token timeout) it gives up and proposes a new ring without the node it can't hear. The others then have 15.3 seconds (the consensus timeout) to agree on the new membership, and they carry on without the stuck node.

That works well when the token has disappeared. It works badly when the token is just late. A token that turns up after 10 or 11 seconds resets the clock, so the cluster never gets as far as removing anyone, even though it's barely working.

The watchdog

Each hypervisor runs an HA agent that looks after the VMs on that node. Roughly every 10 seconds the agent writes its state to the shared cluster filesystem, which itself travels over corosync. Every successful write resets a 60-second watchdog timer.

If the agent goes 60 seconds without a successful write, the watchdog expires and the node reboots itself. We call this self-fencing. It's there to make sure two nodes never both think they own the same VM and end up running it twice.

The watchdog has no idea why the writes stopped. Losing quorum is the obvious reason, but that isn't what got us. Every node still had quorum; the shared filesystem just couldn't make progress because cluster messaging was stuck. The watchdog treats both cases the same way and reboots the node.

It's worth keeping the two timers apart. The token timeout is the cluster deciding to give up on a member. The watchdog is a node deciding to give up on itself. On 5 October the watchdogs got there first.

4. The fault on sg07

sg07 is an AMD EPYC Turin node. Over several hours the DIMM in slot P0MG0 threw more and more corrected memory errors, peaking at 47 a minute around 02:00. None of them crashed the node, but each one froze it for up to half a second.

Corrected errors still cost time

Server memory is protected by ECC (Error-Correcting Code). If a bit is misread, ECC rebuilds the right value and the read goes through, so software never notices. These are corrected errors, and one on its own is nothing to worry about. This is what they looked like in sg07's kernel log:

sg07 kernel: mce: [Hardware Error]: Machine check events logged
sg07 kernel: [Hardware Error]: Corrected error, no action required.
sg07 kernel: [Hardware Error]: CPU:24 (1a:2:1) MC22_STATUS[Over|CE|...]
sg07 kernel: [Hardware Error]: FRU Text: Locator: P0MG0
sg07 kernel: EDAC MC0: 1 CE on mc#0csrow#0channel#9 ...

The trouble is in how they're handled. sg07 uses firmware-first error handling, so for every error the platform firmware takes over the CPU, deals with it and hands control back. The operating system is simply paused in the meantime and has no idea it happened. At the peak, corosync on sg07 logged scheduling pauses of 381 – 541 ms, where we'd normally see single-digit milliseconds. At 47 errors a minute, sg07 was frozen for something like a third of every minute. A handful of these an hour is fine. Close to one a second isn't.

The error rate climbed for hours

The jump at 01:49, from about 12 to about 32 errors a minute, came 29 minutes before the outage, and by 01:57 Ceph was already responding slowly. We missed both. We had no alert on corrected-error rate, and the rise in IO wait wasn't big enough to cross our thresholds.

Only sg07 degraded

You can see it in how long each node took to finish its periodic status collection. sg07 got steadily slower, while every other node stayed where it normally sits:

We didn't find any hardware or network problem on the other 16 hypervisors. They stayed reachable, ran their VMs normally right up until they rebooted, and logged no errors of their own.

5. The cascade

sg07 stalled the token twice. Both times it got moving again with a second or two left on the 12.75-second clock, so none of the healthy nodes ever got as far as dropping sg07 from the ring. The shared filesystem wasn't getting updates through in the meantime, though, and 60 seconds after the last one landed, the watchdogs started going off.

Second by second

Why the cluster didn't recover in time

A missing token on its own can't stop the cluster recovering, because removing a stuck member is triggered by the token timeout and done over normal cluster messages. A token that keeps arriving late can, and that's what happened here. The 16 healthy nodes logged the 9.5-second warning five times between 02:16:25 and 02:19:07, and not one of them made it to 12.75 seconds before its watchdog went off.

The only node that did reach the limit was sg07 itself, at 02:18:36. So in the end it was the faulty node that asked to be removed, and the new ring formed about two seconds after the first two watchdogs had already fired.

From there, each wave of fences pulled members out of a ring that had only just formed, which kicked off another 12.75 s + 15.3 s reform. Membership never settled long enough for any node to write to the shared filesystem and reset its watchdog. There was nothing wrong with the healthy nodes. They rebooted because they were stuck waiting on sg07.

6. Why VMs were slow and unreliable to restart

Between 03:17 and 10:28 there were 6,503 forced takeovers of disk locks. Some of those takeovers cut off hypervisors that were perfectly healthy, which led to more takeovers. That loop went on for hours, and it's why ten hypervisors had to be rebooted a second time.

How disk locks work

Every VM disk is a Ceph RBD image, and a hypervisor has to hold an exclusive lock on it, stored in Ceph, before it can write. When a VM moves normally, the lock is handed over cleanly.

After the reset, every running VM's lock still belonged to a hypervisor process that no longer existed. Ceph won't just drop a lock, since that could let two hypervisors write the same disk. Instead the new owner blocklists the old one, so the storage servers refuse anything further from it, and then takes the lock. If the old owner really is gone, that's harmless.

How healthy hypervisors got cut off

The catch is how Ceph decides the old owner is gone. It checks whether the owner still has an open connection to the disk. During recovery the storage daemons were restarting as well, so every hypervisor kept having to reconnect, and for a few seconds at a time a healthy hypervisor with running VMs looked exactly like a dead one. If a VM started somewhere else in that gap, it took the lock and blocklisted the healthy hypervisor. Disks left mapped after failed VM starts made this happen more often.

It spread because each hypervisor uses a single Ceph client for every disk it has mapped, and a blocklist applies to the whole client. Blocklisting a hypervisor over one disk took out every VM on it. Their writes failed, they got restarted on other nodes, and those restarts triggered more takeovers, some of which landed on other healthy hypervisors.

A blocklisted client doesn't come back by itself. On each affected hypervisor we unmapped the leftover disks and tested. When that wasn't enough, we rebooted the node so it reconnected to Ceph as a fresh client. Ten of them needed that.

What the log messages mean

7. Why restoration took six hours

The failure itself took about three minutes, from the first stall at ~02:16:15 to the last fence at 02:19:29. Getting everything back took six hours, mainly for four reasons.

7.1 Three hypervisors didn't boot (81 – 92 minutes)

14 of the 17 hypervisors came back on their own within 1.5 – 3.5 minutes. The other three got stuck at the boot screen. We could reach their BMCs, but the remote KVM console wouldn't take any input, so there was nothing more we could do remotely. Datacenter staff stepped in, and the three nodes came back at 03:40, 03:47 and 03:51.

We're still working out why they hung. It doesn't look related to the memory fault, since these were different nodes that fenced normally along with the rest.

7.2 Storage recovery was slow

With all 17 nodes rebooting at once, Ceph lost a large number of OSDs together and had to rebuild redundancy. Low free space held that back:

Low space hindering backfill (add storage if this doesn't resolve itself):
55 pgs backfill_toofull

55 placement groups couldn't backfill because the OSDs they needed to move to were too full. On top of that, plenty of OSDs were still offline and PGs were waiting on replication.

7.3 Stale storage locks

Locks still held by clients that no longer existed stopped volumes from attaching cleanly, and clearing them is what blocklisted healthy hypervisors (section 6). Sorting it out meant unmapping disks and, on ten hypervisors, a second reboot.

7.4 Around 1,800 VMs had to be started slowly, on purpose

The HA manager starts any VM it thinks should be running, and after a full outage that's all of them. Turning HA back on would have tried to start ~1,800 VMs at once, which would have saturated storage and could easily have caused a second outage. So instead we:

1. Kept HA disabled.
2. Cancelled the start tasks that had queued during the outage.
3. Set ~1,229 HA-managed VMs to "stopped" so HA would not start them later.
4. Started VMs manually in batches of 10 – 20, node by node, checking storage load after each batch.

Letting everything start at once would have been quicker, but we weren't willing to risk a second outage to save a few hours.

Some VMs did come back between 03:18 and 03:20, started by HA on its own before we'd disabled it. We hadn't checked any of them, and they were stopped again as part of the controlled restart, so we don't count them as restored. The first VM we started on purpose and confirmed was working was a test workload at 04:28 – 04:29, and batch restoration started at 04:30.

8. Corrective actions

The main thing we're taking from this is that one bad hypervisor shouldn't be able to take down the whole cluster, and getting back up shouldn't depend on all 17 hypervisors and the storage cluster recovering together.

8.1 Separate storage from compute

Right now every node is both a hypervisor and a Ceph storage node, so when a hypervisor fences, its OSDs go down with it. On 5 October that meant fencing 17 hypervisors also took out the entire storage cluster. A compute failure turned into a storage recovery, with backfill stuck on full OSDs, daemons reconnecting, and the lock takeovers that blocklisted healthy hypervisors.

We're going to move Ceph onto dedicated storage nodes, with the hypervisors acting only as Ceph clients. Once that's done:

• A hypervisor that fences only takes its own VMs down. No OSDs go offline and no data has to move.
• Even if every compute node goes at once, storage stays up and healthy, so recovery is just starting VMs. Most of the six hours in section 7 wouldn't have happened.
• Storage nodes don't run the HA watchdog, so a coordination problem on the compute side can't reboot them.
• A hardware fault on a compute node won't slow down OSD peers or client IO anymore.

It isn't free. We'll need more servers, rack space and power, and all VM disk IO will go over the network. We're also looking at breaking the 17-node hypervisor cluster into smaller clusters on shared storage, so a coordination failure like this one affects fewer nodes. We'll share a design and migration plan separately.

8.2 Move HA decisions and fencing outside the hypervisors

Today each hypervisor makes its own call. If it can't confirm cluster state for 60 seconds, it reboots. That stops a VM ever running twice, but it also means a coordination stall gets every node to fence itself, the 16 healthy ones included.

We're going to pull that decision out of the hypervisors. An HA controller running outside the hypervisor cluster will decide which node has failed and fence that one node through its BMC, powering it off over IPMI or Redfish. Hypervisors won't reboot themselves just because coordination has stalled. VMs will keep running through a stall until the controller has evidence that a specific node is faulty, and then only that node gets fenced.

The controller has to be highly available itself and sit in a separate failure domain. It replaces the current watchdog-based self-fencing, so we'll design and test it carefully before it goes into production.

Appendix: timing values and terms