From one node to a cluster: what got better, and what got worse

For about a year, everything here ran on one machine: an AMD Ryzen Threadripper 3960X with 64GB of memory and seven SSDs. There was a post on this blog arguing that one well-specified node beats a vague fleet, and I still think that was right at the time.

It is no longer what happens. Here is what changed, what improved, and the two things that got measurably worse, because a migration post that only lists wins is a press release.

What it is now

A Proxmox cluster, all in the Netherlands:

Node	Chassis	CPU	Memory	Role
pve2	Dell PowerEdge R640	2x Xeon Gold 6136, 24c/48t	128GB ECC	Game servers, billing, VirtFusion
pve3	Dell PowerEdge R640	2x Xeon Gold 6136, 24c/48t	96GB ECC	Game servers
pve4	Custom build	Threadripper 3960X, 24c/48t	32GB	Storage only

The Threadripper did not get thrown away. It now does one job: it holds the ZFS pool and serves it to everything else over NFS on a 10Gbit network that has no uplink of its own.

The full table lives on the specifications page and is kept current.

What got better

ECC memory. This is the one I care most about. Every game server host now runs registered ECC memory. A bit flip in a world chunk on a machine without ECC is written to disk silently and then faithfully replicated by ZFS into every backup, because ZFS checksums protect against the disk lying to you and not against the memory doing it. ECC catches and corrects single-bit errors before they reach the pool. For a host whose entire pitch is not losing your world, that was the weakest link in the chain.

Capacity. From 64GB in one box to 352GB across three usable hosts. That is the difference between turning people away and having somewhere to put them.

Compute redundancy. This is the real change. If a machine dies now, its guests can be started on another one. That was flatly impossible before, and the previous version of this post said so in as many words: "A provider with a hundred nodes migrates you to another one. There is no other one here." There is now.

Remote management. iDRAC and iLO mean a hung host can be power-cycled and reinstalled without someone standing in front of it. On the old machine, a kernel panic meant a physical trip.

Hot-swap and redundant power. Enterprise chassis things that sound boring and matter at 02:00.

What got worse

Per-core speed

This is the honest cost and it is not small.

CPU	Turbo	Generation
Threadripper 3960X	4.5 GHz	Zen 2
Xeon Gold 6136	3.7 GHz	Skylake-SP

Minecraft's tick loop is single-threaded. The number that decides whether your server holds 20 TPS is how fast one core is, not how many there are. Zen 2 and Skylake-SP have broadly comparable IPC, so moving from 4.5 GHz to 3.7 GHz is roughly an 18% reduction in per-core throughput.

I made that trade deliberately. ECC, capacity and redundancy were worth more to me than peak clock, and a server that occasionally ticks a millisecond slower is a better outcome than a server whose host has no spare. But it is a trade, not an upgrade in every direction, and anyone telling you their migration improved every number is not counting carefully.

Storage became a single point of failure

The old failure story was simple: one machine, and if it dies everything stops. Honest, and easy to reason about.

The new one is more subtle and in one respect worse. Compute is now redundant, but all storage lives on pve4. If that host goes down, every node loses its disk at once. The cluster survives losing a compute node; it does not survive losing the storage node.

raidz2 still covers two simultaneous drive failures, and that has not changed. What has changed is that there is now a whole machine in the path that has no redundancy behind it.

Two mitigations, neither complete. The backups are on a separate system and include an encrypted off-site copy, so the data survives the host. And the pool is on SATA SSDs in a standard chassis, so recovery means moving drives rather than waiting for an exotic part. The honest statement is that a pve4 failure is an outage measured in hours, not minutes, and the fix for that is a second storage host rather than a sentence on a website.

The migration is not finished

Every server created now goes onto the R640s, so what is described above is what you get if you order today. A handful of older servers predate that hardware and are still on a machine being emptied, and those are moving across as capacity allows.

I mention it because "we upgraded" usually gets written the day the first new box is racked, and then quietly covers a year of everything not actually being on it. The specifications page lists what is in service rather than what is planned, and if your server is one of the ones still waiting, asking will get you a straight answer.

Why not overselling still caps the business

This part has not changed, and it is still the reason the price is what it is.

The rule is that memory sold to a customer is committed to that customer on the host their server runs on, and counted. When a host is full, it stops taking servers. That caps how many customers fit, and the fixed costs, the hardware, the fibre, the IP space and the power, divide across fewer of them.

A budget host sells the same gigabyte to several customers on the assumption that not everyone is busy at once. That assumption holds until 21:00 on a Friday, when it does not, and your tick rate falls for reasons you cannot see because the cause is on somebody else's server.

The figure on the home page shows how much is currently committed. A host who oversells cannot publish that number, because it would exceed 100%.

What this does not change

Support is still one person, roughly 07:00 to 00:00 CET. More hardware does not add a night shift, and it would be a strange thing to claim it did.

The specifications are still exact and still named per machine, which is the part I would not give up. "Enterprise-grade hardware" is what you write when you do not want the comparison. Hostnames and their CPUs invite it.