100 Concurrent Spark Jobs, 67 Seconds, One $2/Hour Pool
In The Small Job Tax we made an argument: most data platforms make you pay cluster-sized overhead for container-sized work, and your smallest jobs are quietly your most expensive per byte.
An argument is cheap. So we measured it.
One hundred Spark jobs, submitted concurrently, each one scanning ~200 MB of Iceberg data — 19 GiB of real columnar IO and 410 million rows in total — finished in 67.5 seconds, on one small compute pool that costs about $2 an hour. One hundred submitted, one hundred succeeded, zero failures.
Then we kept pushing: 200 jobs, 300, 400, 500. The pool never became the problem.
What exactly is "a job" here
Every number in this post comes from a real, tracked job — submitted through our control plane's job API (the same path a scheduler uses), launched in its own container, recorded on the job monitoring page, and verified from the engine's own counters. No synthetic no-op "SELECT 1" tricks in the headline claim.
Each job:
- is submitted to
POST /job-runs/submit— one HTTP call, like Airflow or Dagster would make; - gets its own lightweight launcher container on AWS Fargate — not a cluster;
- connects to a shared, already-warm Spark compute pool (tier S, 3 workers);
- scans a 199 MB Iceberg table — all 23 columns, 4,098,777 rows — and computes a checksum aggregation over every column;
- reports back, and appears on the monitoring page with live status and duration.
We deliberately made the query read the entire table. Our first version aggregated two columns, and Iceberg's column pruning — doing exactly its job — cut the physical scan to 12 MiB. Honest benchmarking means the workload has to match the claim, so we rewrote it to touch every column and then measured the scan from the engine's own scan-operator metrics: 200,487,731 bytes per job. Not estimated from file sizes. Counted by the operator that did the reading.
The headline result
| per job (measured) | × 100 concurrent | |
|---|---|---|
| Data physically scanned | 191.2 MiB | 19.1 GiB |
| Rows processed | 4,098,777 | 410 million |
| Execution time | 11.5 s isolated · 11.2 s median under full load | — |
| Outcome | — | 100/100 succeeded · 0 rejected · 67.5 s wall clock |
Two details in that table deserve a second look.
The median job under 100-way concurrency ran in 11.2 seconds — statistically identical to a job running alone (11.5 s). One hundred simultaneous arrivals did not degrade individual jobs; the pool absorbed them.
Wall clock was 67.5 seconds for the whole batch. For contrast: provisioning a fresh Spark cluster for a single job — the standard model on cluster-per-job platforms — took about 2.5 minutes on the same infrastructure when we measured it. The entire hundred-job batch finished in less than half the time one cold cluster takes to boot.
We went looking for the ceiling
One clean run at one level proves little, so we ramped: 100, 200, 300, 400, 500 concurrent submissions, all through the tracked control-plane path, all against the same three-worker pool.
| concurrent jobs | succeeded | wall clock | sustained throughput |
|---|---|---|---|
| 100 | 100 / 100 | 69.5 s | 86 jobs/min |
| 200 | 200 / 200 | 118.4 s | 101 jobs/min |
| 300 | 298 / 298 launched¹ | 212.6 s | 84 jobs/min |
| 400 | 400 / 400 | 202.3 s | 119 jobs/min |
| 500 | 470 / 470 launched¹ | 228.7 s | 123 jobs/min |
Every job that launched, succeeded — 1,468 of them across the ramp, zero execution failures. And throughput was still rising at 500: the pool digests a deeper queue more efficiently, because task slots stay saturated. We did not find the execution ceiling; we ran out of afternoon.
¹ Full disclosure: the handful of submissions lost at the 300 and 500 levels were rejected before reaching the platform — our control plane for this test ran on a Windows development workstation whose network stack gave out under hundreds of simultaneous connections. The pool, the launcher, and AWS never refused anything. We are stating this because the alternative — quietly renaming 470 to "100%" — is how benchmark posts lose your trust.
The 100-job burst used roughly 1.25% of a default AWS account's Fargate capacity quota. The constraint in this architecture is the pool — and the pool is the cheap, boring thing you scale.
Why this works: containers for jobs, pools for compute
The cluster-per-job model — Databricks job clusters, EMR steps, spark-submit against fresh infrastructure — makes every small job pay a provisioning tax measured in minutes. Serverless SQL warehouses (Databricks SQL Serverless, Trino-based services) solve the latency but meter you on bundled platform pricing, and they are query engines — not a general path for your Python and JAR jobs.
Cazpian splits the difference:
- A warm compute pool holds Spark capacity. It is shared, always ready, and costs what a few Fargate containers cost — our tier-S pool with three workers runs about $2.005/hour in raw AWS infrastructure (us-east-1, on-demand, August 2026 rates).
- A thin launcher container per job provides isolation, its own logs, its own exit code, its own row on the monitoring page. Launch takes ~20–27 seconds — and that stayed flat from 1 job to hundreds, because placing small containers is what ECS is built for.
- Native Query Acceleration does the scanning. The 200 MB-per-job figure in this post is read straight from its scan operator's counters, and the acceleration is why 16× more IO per job (191 MiB vs our first 12 MiB version) made no difference to the wall clock — at this job size, the engine is fast enough that orchestration dominates, not IO.
The economics land where the Small Job Tax post predicted. The 67.5-second run consumed about four cents of pool time; add the hundred launcher containers and the whole thing cost roughly seven cents of infrastructure — for 19 GiB scanned and 410 million rows processed. Run it every hour, every day for a month: about $50. The same hundred jobs on per-job clusters pay the provisioning tax a hundred times per batch — on our measurements, that model costs on the order of 150× more per job for work this size, before you even count the 2.5 minutes of latency per job.
A fairness note: our figure is raw AWS infrastructure; serverless competitors bundle their platform into a per-unit price. Compare accordingly — but also ask what fraction of a bundled price is doing 200 MB of work.
What broke along the way (and why you should care)
We found the edges of our own platform during this test, fixed them, and re-ran. Three examples, because a benchmark that finds nothing wasn't pushing:
- A permissions cache needed to be single-flight. Under a concurrent burst, every submission missed the cache at the same instant and stampeded an AWS rate limit — rejecting ~1–5% of submissions. A cache alone wasn't enough; only one caller may fill it.
- A throttled status poll must not fail a healthy job. At one point 35 of 100 running jobs were marked FAILED while their containers finished successfully — a transient AWS throttle was being treated as a terminal answer. Transient now means retry, and 400/400 clean followed.
- Durations must come from the task, not the observer. Our control plane originally stamped timestamps when its poller noticed state changes — recording a 28.8-second execution as 1.1 seconds. Every duration in this post comes from the task's own clock, verified to the millisecond against the engine.
Each of these is invisible at 10 concurrent jobs and unmissable at 100. If your platform hasn't been tested at this concurrency, these bugs — or their cousins — are probably in it.
What we did not test
One workload shape (scan + aggregate; no joins, no writes, no shuffling monsters). One run per ramp level, not a distribution. A single tier-S pool — no autoscaling. And the 200 MB per job is the same table read by every job, served hot — your S3 cold reads will add latency the first time through. The TPC-DS terabyte benchmark covers the heavy end; this post is deliberately about the small end, where the tax lives.
Try the math on your own platform
Pull your job history. Count the jobs under 1 GB. Multiply by your per-job provisioning overhead — in minutes and in dollars. That number is what a warm pool with per-job containers makes go away.
One hundred jobs. Two hundred megabytes each. Sixty-seven seconds. Seven cents.
The small job tax is optional.