Benchmark: are ARM runners worth migrating to?
Three weeks of running every pipeline twice, on x86 and ARM, across four real repositories. The answer was not uniform — it depended entirely on the workload.
The common claim is that moving CI runners to ARM cuts roughly forty percent of the bill. We measured it across four production repositories over three weeks, running every pipeline on both architectures simultaneously.
Method
Every commit to the default branch ran twice, on equivalently sized machines with four vCPUs and sixteen gigabytes of memory. No cache was shared between architectures.
mizban ci benchmark --repos payments,ledger,web,ingest \
--arch amd64,arm64 --runs 240 --output csvResults
Median total pipeline duration, in seconds:
| Repository | Language | x86 | ARM | Change |
|---|---|---|---|---|
| payments | Go | 142 | 119 | −16% |
| ledger | Rust | 406 | 341 | −16% |
| web | TypeScript | 218 | 237 | +9% |
| ingest | Python | 95 | 168 | +77% |
What explains the spread
The repositories that compile native code — Go and Rust — benefited. The TypeScript repository got slightly slower, because most of its wall clock is spent in single-threaded work where ARM has no clock advantage.
The disaster was ingest. It depends on numpy and two other libraries that at the time shipped no prebuilt manylinux_aarch64 wheels. pip compiled them from source on every run.
The number in the cloud vendor's benchmark was measured on a workload somebody had already tuned for ARM. Yours is not that workload until you prove it is.
What we did
The first three repositories migrated. ingest stayed on x86 until its dependencies publish wheels. The real saving was neither forty percent nor zero — about nineteen percent of total CI spend, which was worth the effort.
The methodological point matters more than the number. Had we benchmarked only payments, we would now have a completed migration and one permanently broken pipeline. A benchmark has to include your worst workload, not your favourite one.