We Moved Cazpian's Spark Pools to ARM — 2x Faster Under Load

Conventional wisdom says migrating a data engine to ARM is a performance gamble you take for the discount. We ran the experiment on our own Spark compute pools, and the result inverted the premise: under 100-way concurrency, the ARM64 pool ran every job roughly twice as fast as its x86 twin — at about 20% lower infrastructure cost.

We are publishing the numbers, but also the part benchmark posts usually cut: our first measurement said ARM was 8.5x slower, and that number was wrong in a way that had nothing to do with ARM. If you are planning an ARM migration for your data platform, the detour is probably worth more to you than the table.

The experiment

Two Spark compute pools on AWS Fargate, deliberately identical except for the chip:

	x86 pool	ARM64 pool
driver	4 vCPU / 16 GB	4 vCPU / 16 GB
workers	3 × 8 vCPU / 60 GB	3 × 8 vCPU / 60 GB
Spark / Iceberg	same versions	same versions
Native Query Acceleration	active, verified in the plan	active, verified in the plan

The workload is the same one from our 100-concurrent-jobs benchmark: each job scans all 23 columns of a 199 MB Iceberg table (4.1 million rows) and computes a checksum aggregation — 100 jobs submitted concurrently through the control plane's job API, the same path a scheduler uses. We ran four bursts of 100, alternating which architecture went first to cancel warm-up effects.

"Verified in the plan" is doing real work in that table. Native acceleration fails soft: if the fast path cannot engage, the query silently falls back to standard Spark and still returns correct results. A pool in that state looks completely healthy. We checked the executed plans on both drivers before trusting a single timing — this paragraph is foreshadowing.

The results

Reliability first: 400 of 400 jobs succeeded, on both architectures, in every round.

Per-job execution time, 200 jobs per architecture:

	x86	ARM64
median (p50)	9,953 ms	5,109 ms
p90	16,149 ms	6,646 ms
mean	10,171 ms	6,388 ms

And per round, so you can see it is not an averaging artifact — ARM won both rounds, in both orders:

per-job p50	round 1	round 2
x86	10,496 ms	9,682 ms
ARM64	5,942 ms	4,680 ms

The engine's own SQL timer tells the same story: roughly 2.4 s median on ARM versus 6.0 s on x86 under full load. Interestingly, a single job run in isolation is a tie — about 2.8 s on either chip. The ARM advantage appears specifically under concurrency: its latency distribution barely widens as 100 jobs pile onto the pool, while x86's roughly doubles. Neither pool was CPU-saturated (ARM peaked around 22% of its reservation), so this looks like the sustained-throughput behavior Graviton is known for rather than a headroom effect — we state that as a reading of the data, not a microarchitectural proof.

The cost line

Straight from public AWS Fargate pricing (us-east-1, on-demand, September 2026): this pool shape — 28 vCPU and 196 GB across driver and workers — costs about $2.00/hour on x86 and $1.60/hour on ARM64. That is the standard ~20% Graviton discount, and it now arrives with a per-job speedup instead of a tax.

The metric we threw out

Batch wall clock — first submit to last completion — ranged from 67.7 s to 98.7 s across the four rounds, and its best single round happened to be x86's. We excluded it from the verdict, and you should exclude it from yours: at this job size, wall clock measures how fast the orchestrator places 100 launcher containers, a pipeline that is identical (and, in our case, still x86) for both pools, and that varies ±40% run to run on its own. A batch whose median job takes 5 seconds posting a 96-second wall is telling you about container placement, not about the chip. If you A/B architectures on batch wall at small job sizes, you are benchmarking your scheduler.

The 8.5x detour, or: it is never the chip

Our first A/B produced a headline we very nearly believed: ARM64 was 8.5x slower on the engine's own clock. It took most of a day to discover that number was true, measured honestly — and meaningless. Three layers had to be peeled.

Layer one: the ARM pool was never accelerated. Our platform approves native acceleration per image digest, one digest per repository — and the x86 and ARM builds of the Spark image share a repository. Approving one silently un-approved the other. The ARM pool started, passed health checks, answered queries, and ran every scan on the slow path. The 8.5x compared accelerated x86 against unaccelerated ARM. The fix was making approval accept multiple digests; the lesson was older than the fix: read the executed plan before trusting any acceleration number.

Layer two: with acceleration approved, every ARM scan failed outright — a native S3 credential error. Then the same build's x86 sibling failed identically, which exonerated ARM and indicted the image. We diffed the two images down to the bone: all 300+ jars byte-for-byte identical, environments identical, every configuration delta tested key-by-key on live pools. Nothing explained it. Because nothing in the images was the cause.

Layer three: the timeline. Every "working" measurement predated — and every failure postdated — a cost-saving scheduler in our dev cluster stopping one background service at noon. That service turned out to sit at the bottom of a credential chain: the control plane delegated secret decryption to it, the catalog service therefore dropped the S3 credentials from the catalog it hands to Spark pools, and the native scan — whose only credential source is that hand-off — failed on every read. The standard JVM path falls back to the pool's IAM role, so ordinary queries kept working and every pool kept looking healthy. Three independent fail-softs composed into one hard failure, four services away from its cause.

We re-tested the "good" old image during the outage: it failed identically. We restored the stopped service: the "broken" new image ran at full speed. Both images were always fine. When two provably identical artifacts behave differently, stop diffing the artifacts and start diffing the timeline.

The checklist we wish we'd started with

1. Verify the fast path in the executed plan, per architecture, before comparing anything. Fail-soft acceleration makes "healthy but silently slow" the default failure mode.
2. Run your control at the same time as your candidate. Our x86 "baseline" was measured hours before the ARM runs, across an infrastructure change nobody clocked. Same-time controls would have collapsed the whole investigation to minutes.
3. Alternate execution order and run multiple rounds. Our ARM advantage held in both orders; that is what let us publish it.
4. Separate orchestration metrics from compute metrics. Per-job and engine-side timings compare chips; batch wall clock compares schedulers.
5. Audit what happens to your pools on restart. We found (and are fixing) a path where a restarted ARM pool silently re-registers as x86 — the kind of regression that only shows up in the plan, per lesson one.
6. Know your credential chain end to end. The native engine's S3 client did not fall back to the task role the way the JVM does. Whatever vends credentials to your fast path is part of your query path's availability.

The 1 TB rematch

Concurrent small jobs are one workload shape. Before calling the migration, we re-ran our TPC-DS 1 TB benchmark as a same-night A/B: two freshly created pools, four workers and 64 worker vCPUs each, the same build of our runtime on both sides, only the architecture differing. All 103 queries, one shot each, no warm-up, zero failures on either side.

	x86	ARM64
Total, 103 queries	4,494 s	2,926 s
Median query	30.6 s	20.3 s
Slowest query	266.6 s	125.6 s
Queries faster	—	102 of 103

ARM64 ran the full sweep 1.54× faster — and did the same work with 1.8× less executor CPU time and half the garbage collection, which is the honest chip-level signal underneath the wall clock.

Every query, both architectures

Summaries hide things, so here is the whole sweep — one execution per query, wall clock, same night, same build, the only difference being the chip. Speedup above 1.00× means ARM64 was faster.

query	x86 s	ARM64 s	speedup	query	x86 s	ARM64 s	speedup
q1	14.6	10.8	1.35×	q49	28.9	18.5	1.56×
q2	20.3	16.4	1.24×	q50	59.5	32.4	1.83×
q3	17.8	12.4	1.43×	q51	41.0	28.2	1.46×
q4	177.3	111.4	1.59×	q52	15.3	10.5	1.46×
q5	35.7	25.4	1.40×	q53	18.3	13.9	1.32×
q6	17.2	11.1	1.55×	q54	36.6	20.8	1.76×
q7	33.5	23.9	1.40×	q55	13.9	9.7	1.43×
q8	13.8	10.9	1.26×	q56	30.6	22.0	1.39×
q9	71.3	52.3	1.36×	q57	19.4	12.9	1.50×
q10	45.9	34.3	1.34×	q58	40.5	27.8	1.46×
q11	106.8	68.6	1.56×	q59	39.8	30.3	1.31×
q12	8.9	6.9	1.29×	q60	34.5	25.1	1.38×
q13	19.5	13.6	1.44×	q61	72.4	40.1	1.80×
q14a	105.2	67.2	1.56×	q62	8.7	6.2	1.39×
q14b	108.5	69.1	1.57×	q63	17.2	12.2	1.42×
q15	33.7	21.0	1.60×	q64	135.4	82.9	1.63×
q16	29.8	20.0	1.49×	q65	46.5	30.1	1.54×
q17	51.5	33.7	1.53×	q66	20.8	14.1	1.47×
q18	30.5	18.9	1.61×	q67	150.9	86.3	1.75×
q19	44.9	27.0	1.66×	q68	33.4	20.3	1.64×
q20	12.4	9.1	1.36×	q69	32.6	24.7	1.32×
q21	4.6	3.6	1.28×	q70	32.3	19.9	1.62×
q22	17.3	13.6	1.27×	q71	34.5	23.7	1.45×
q23a	153.9	104.3	1.48×	q72	266.6	125.6	2.12×
q23b	160.2	104.9	1.53×	q73	13.2	8.3	1.60×
q24a	108.8	65.3	1.67×	q74	82.4	55.6	1.48×
q24b	111.0	67.3	1.65×	q75	113.1	87.1	1.30×
q25	45.4	31.8	1.43×	q76	26.2	19.7	1.33×
q26	12.3	9.7	1.26×	q77	27.0	21.3	1.26×
q27	26.9	18.8	1.43×	q78	124.4	87.2	1.43×
q28	64.4	44.9	1.44×	q79	31.9	22.4	1.42×
q29	52.4	33.8	1.55×	q80	58.0	38.5	1.50×
q30	11.0	8.0	1.37×	q81	11.0	7.6	1.45×
q31	88.2	58.0	1.52×	q82	38.8	23.7	1.64×
q32	12.8	9.2	1.39×	q83	5.4	4.2	1.26×
q33	33.9	23.6	1.44×	q84	4.3	3.0	1.40×
q34	14.0	8.9	1.57×	q85	25.0	19.3	1.29×
q35	35.3	27.4	1.29×	q86	5.5	4.5	1.22×
q36	20.5	13.2	1.56×	q87	69.6	44.4	1.57×
q37	23.6	15.7	1.51×	q88	53.7	33.7	1.59×
q38	70.9	44.6	1.59×	q89	22.6	17.7	1.28×
q39a	10.9	7.3	1.50×	q90	4.4	4.9	0.90×
q39b	9.6	5.5	1.74×	q91	7.9	5.7	1.39×
q40	13.9	9.7	1.44×	q92	6.8	6.4	1.06×
q41	1.8	0.7	2.49×	q93	70.3	38.2	1.84×
q42	15.6	9.0	1.73×	q94	18.9	14.5	1.31×
q43	17.5	11.1	1.58×	q95	59.3	40.3	1.47×
q44	20.8	15.7	1.33×	q96	8.2	5.4	1.51×
q45	21.5	13.6	1.58×	q97	43.9	30.3	1.45×
q46	35.1	22.1	1.58×	q98	33.2	22.6	1.47×
q47	51.0	33.4	1.53×	q99	13.2	8.8	1.49×
q48	22.2	13.6	1.63×

One hundred and two of the hundred and three land above 1.00×. The lone exception is q90 at 0.90×, and the biggest wins cluster exactly where the work is heaviest — q41 at 2.49×, q72 at 2.12×, the multi-terabyte sorts all north of 1.4×.

That result moves the comparison we published in the 1 TB post. On the same 99-query basis and against the same third-party-measured Trino figure (2,552 s), our x86 run stood 1.49× slower. The ARM64 run comes in at 2,679 s — within 5% of Trino, on compute that costs a fifth less. The caveats from that post still apply — published figure, different hardware, local NVMe on their side — and the only unarguable comparison remains running both engines ourselves, which is now firmly on the list. But the trend line is the story: stock Spark measured 3.5× slower than Trino on identical hardware; Native Query Acceleration brought us to 1.49×; the same acceleration on ARM64 brings us to 1.05×, at lower cost, without giving up Spark's ecosystem or open Iceberg tables.

What we did not test

Python-UDF-bound jobs get their own A/B before we migrate them. Both benchmark nights ran on a dev cluster, not a production distribution, and single-shot queries carry no confidence intervals. Our launcher containers stayed x86 throughout — per-job container placement was identical for both pools by construction.

The verdict

ARM64 stays. Same reliability, roughly half the per-job latency under the concurrency that actually matters to us, a 1.54× faster 1 TB benchmark that lands within 5% of the published Trino figure, and a fifth off the infrastructure bill — with the small-job economics we keep writing about, the discount compounds on every one of those hundred containers. The gamble conventional wisdom warned us about turned out to be the status quo.