What a Terabyte Actually Costs: TPC-DS on Cazpian
There is a long-standing assumption in data engineering: if you want fast SQL, you do not use Spark. Spark is for pipelines. For interactive analytics you bring in a dedicated query engine, run it alongside, and accept the cost of operating two systems over one set of tables.
We wanted to know how true that still is. So we ran the full TPC-DS benchmark — all 103 queries — against a terabyte of Apache Iceberg data on a four-worker Cazpian cluster, and published everything: every query time, the cost, the hardware utilisation, and the parts that are not yet good enough.
The short version
| Queries completed | 103 of 103, no failures |
| Total time | 3,813 seconds (63 minutes) |
| Median query | 29 seconds |
| Cost of the entire run | $6.49 |
| Time to generate the 1 TB dataset | 51 minutes, $2.20 |
For anyone who does not spend their days reading query plans: we asked a modest four-machine cluster ninety-nine of the hardest analytical questions the industry has standardised on, against a terabyte of data, and it answered all of them for the price of a sandwich.
What we actually ran
TPC-DS is the industry-standard benchmark for decision-support workloads — the kind of SQL that powers dashboards, finance reporting and customer analytics. It is deliberately unpleasant: deep joins, correlated subqueries, window functions, and queries that touch billions of rows.
The data. Scale factor 1000 — universally called "the 1 TB dataset". That terabyte is the raw generated volume; written as compressed Parquet in Iceberg it occupies 236.9 GB across 2,733 files averaging 88.8 MB each, in Amazon S3 with an AWS Glue catalog. 6.35 billion rows across 24 tables.
The hardware. Four worker nodes plus one coordinator:
| per worker | cluster | |
|---|---|---|
| vCPU | 16 | 64 |
| Memory | 120 GB | 480 GB |
| Task slots advertised | 24 | 96 |
| Storage | network-attached ephemeral | — |
The coordinator adds 8 vCPU and 32 GB. Nothing exotic: this is a cluster a mid-sized team would run for a department.
The method. Every query executed exactly once. No warm-up pass, no repetitions, no result caching, no materialised views, no query-specific tuning. We primed the cluster with three trivial statements first so that Java's start-up cost was not charged to whichever query happened to run first — that is the only concession, and it changes no query's result.
No tuning. The tables were written exactly as the generator emitted them. No partitioning strategy, no sort order, no compaction pass, no indexes. A real deployment would do all of those and go faster; we wanted the honest out-of-the-box number.
Every query, every time
This is the table most benchmark posts leave out. All 103 queries, with the wall-clock time each one took — the seconds a person would have waited for the answer:
| Query | Seconds | Query | Seconds | Query | Seconds | Query | Seconds |
|---|---|---|---|---|---|---|---|
q1 | 16.0 | q24b | 95.2 | q49 | 26.6 | q75 | 107.5 |
q2 | 16.1 | q25 | 45.5 | q50 | 54.5 | q76 | 23.8 |
q3 | 15.3 | q26 | 12.1 | q51 | 37.4 | q77 | 25.7 |
q4 | 158.9 | q27 | 24.4 | q52 | 14.1 | q78 | 112.8 |
q5 | 30.3 | q28 | 57.2 | q53 | 16.9 | q79 | 30.9 |
q6 | 14.4 | q29 | 51.2 | q54 | 35.5 | q80 | 56.0 |
q7 | 30.4 | q30 | 11.8 | q55 | 13.3 | q81 | 11.9 |
q8 | 12.5 | q31 | 82.4 | q56 | 28.8 | q82 | 38.0 |
q9 | 58.6 | q32 | 11.8 | q57 | 17.7 | q83 | 5.0 |
q10 | 41.3 | q33 | 29.7 | q58 | 37.2 | q84 | 3.7 |
q11 | 104.3 | q34 | 12.2 | q59 | 36.7 | q85 | 27.4 |
q12 | 8.5 | q35 | 33.6 | q60 | 32.8 | q86 | 5.3 |
q13 | 20.0 | q36 | 18.7 | q61 | 69.4 | q87 | 63.7 |
q14a | 95.3 | q37 | 23.1 | q62 | 8.3 | q88 | 47.6 |
q14b | 98.3 | q38 | 63.5 | q63 | 16.1 | q89 | 22.2 |
q15 | 32.3 | q39a | 11.7 | q64 | 120.3 | q90 | 4.4 |
q16 | 27.5 | q39b | 9.6 | q65 | 44.3 | q91 | 7.8 |
q17 | 50.7 | q40 | 11.9 | q66 | 19.3 | q92 | 6.9 |
q18 | 29.2 | q41 | 0.7 | q67 | 142.2 | q93 | 65.3 |
q19 | 43.1 | q42 | 11.5 | q68 | 32.6 | q94 | 19.4 |
q20 | 11.4 | q43 | 13.3 | q69 | 32.1 | q95 | 59.5 |
q21 | 4.4 | q44 | 17.9 | q70 | 29.0 | q96 | 7.5 |
q22 | 18.4 | q45 | 19.7 | q71 | 34.8 | q97 | 41.1 |
q23a | 150.9 | q46 | 31.8 | q72 | 236.4 | q98 | 32.2 |
q23b | 156.0 | q47 | 46.7 | q73 | 12.0 | q99 | 11.9 |
q24a | 96.3 | q48 | 19.0 | q74 | 80.0 |
Distribution: median 29.0 s · p90 96.3 s · p95 120.3 s · slowest 236.4 s (q72).
Twelve queries finished in under 10 seconds; nineteen took over a minute; six took over two.
How that compares
The honest comparison is against Trino, the most widely deployed open-source engine for exactly this job. The most useful published figure runs Trino on Iceberg on the same four-worker, 64-vCPU shape at the same scale factor, measured by StarRocks:
| 1 TB, 99 queries, 64 vCPU | |
|---|---|
| Trino on Iceberg | 2,552 s |
| Cazpian on Iceberg | 3,813 s |
We are 1.49× slower than Trino. We are publishing that because it is the number, and because anyone can reproduce it in an afternoon.
Now the context that makes it a result worth writing about.
On identical hardware, with no acceleration, an independent evaluation by MR3 measured Apache Spark at 15,678 seconds against Trino's 4,442 — Spark 3.5× slower on the same 99 queries. That gap is the reason for the assumption we opened with, and it is why so many organisations run two engines.
Cazpian's Native Query Acceleration closes roughly 80% of that gap. Instead of 3.5× behind a dedicated query engine, we are 1.5× behind — while remaining Spark, on open Iceberg tables, with the same cluster running your pipelines.
A note on the comparison. Trino's figure was measured by a third party on m6id instances with 950 GB of local NVMe per node. Our workers use network-attached storage. That difference matters — see Where the time goes. We ran 103 queries to their 99; the totals above use the 99-query basis so they are on the same footing. And the only truly unarguable comparison is one where we run both systems ourselves. We intend to.
What Native Query Acceleration is
Traditional Spark processes data row by row through the Java Virtual Machine. Native Query Acceleration replaces the hot paths — scans, filters, joins, aggregations, sorts — with vectorised code that operates on columnar batches outside the JVM, built on open-source Apache Arrow columnar technology.
We wrote up the engineering detail separately, including a version mismatch that nearly made us publish the opposite conclusion: how we benchmarked Native Query Acceleration on Iceberg. On a 200-million-row aggregation over decimal columns — the shape most finance and revenue reporting takes — it runs 4.14× faster through the Iceberg catalog.
For the non-technical reader: it is the difference between reading a spreadsheet one cell at a time and reading a whole column at once.
On this run, 87.3% of query operators executed natively on queries taking more than five seconds; 83.6% weighted by the time they consumed. The remaining fraction falls back to standard Spark, transparently.
You do not configure it, rewrite queries, or use a special table format. It is on.
The cost
Compute cost, computed from public AWS rates and the measured runtime:
| per hour | per run | per query | |
|---|---|---|---|
| Cazpian | $5.19 | $6.49 | $0.063 |
| Cazpian on spot instances | $1.63 | $2.04 | $0.020 |
For comparison, the same hardware shape on Databricks — estimated from their published DBU rates, since they publish no TPC-DS figure at this scale:
| per run (estimated) | vs Cazpian | |
|---|---|---|
| Jobs Compute | ~$10 | ~1.6× |
| SQL Serverless | ~$16 | ~2.4× |
| SQL Pro | ~$19 | ~3.0× |
These are estimates and we have marked them as such. DBU consumption varies by instance type, and rates vary by region, tier and commitment. We encourage you to run the numbers for your own workload rather than take ours — see our compute cost breakdown for how the billing models differ.
The point is not a precise multiple. It is that the price/performance conversation is different from the raw-speed conversation, and for most teams it is the one that matters.
Loading the data
Before you can query a terabyte you have to write one. Generating all 24 TPC-DS tables and writing them as Iceberg took:
| Rows written | 6,347,385,880 |
| Wall clock | 51 minutes |
| Throughput | 2.06 million rows/second |
| Compute cost | $2.20 |
| Files produced | 2,733, averaging 88.8 MB |
Every table landed on its exact specification row count. The average file size is worth noting on its own — 88.8 MB with no tuning and no compaction pass, which means the dataset was immediately queryable rather than needing a clean-up job first.
This is data generation, not ingestion: rows are generated in-cluster, so there is no source system, network fetch or format conversion. Do not read it as a migration estimate.
Where the time goes
For readers who want the machine-level detail, here is what the cluster was actually doing:
| Metric | Value | What it means |
|---|---|---|
| Executor run time | 342,975 s | total time across all task slots |
| Executor CPU time | 157,042 s | of which actually computing |
| CPU efficiency | 0.458 | under half of occupied slot time is compute |
| Shuffle fetch wait | 2,715 s | 65% of wall clock spent waiting for data |
| Shuffle read / write | 1,008 GB / 677 GB | data moved between workers |
| Stages / tasks | 1,203 / 238,198 | units of parallel work |
| Failed tasks | 24 (0.010%) | all retried automatically, no query lost |
| JVM garbage collection | 983 s (0.29%) | negligible |
And the hardware, measured independently through AWS container metrics rather than from Spark's own accounting:
| Resource | Average | Peak | Allocated |
|---|---|---|---|
| Worker CPU | 64% | 100% | 16 vCPU |
| Worker memory | 23% | 53% | 120 GB |
| Worker local disk | 10.6 GB | 41.1 GB | 214.7 GB |
| Disk written per worker | — | 57.7 GB | — |
The headline for an engineer: shuffle wait is 65% of the run. The cluster is not short of CPU — it hits 100% at peak — and it is not short of memory. It spends most of its time waiting for data to move between machines and to and from disk.
That is also the clearest signal of where the remaining gap to Trino lives, and it points at a specific fix: the comparison cluster has 950 GB of local NVMe per node and ours does not. Sorting a terabyte spills to disk regardless of how much memory you give it — we measured 39.9 GB of spill per run — and spilling to network-attached storage costs more than spilling to a local drive. Moving these workloads onto instances with local NVMe is the next thing we are doing.
How many people can use it at once
A single-query benchmark tells you nothing about a Monday morning. We measured concurrency separately, at the 100 GB scale.
Rather than pick eleven convenient queries, we stratified them across the cost distribution of the full suite — three from each speed band, plus the two heaviest in the set. Concurrency pain shows up in the expensive queries, so a subset of fast ones would have flattered the result:
| Band | Single-user time | Queries chosen |
|---|---|---|
| Short | under 5 s | q1 (2.4 s), q55 (3.7 s), q2 (4.4 s) |
| Medium | 5–15 s | q76 (6.5 s), q94 (8.1 s), q65 (11.4 s) |
| Long | 15–40 s | q9 (17.0 s), q93 (22.1 s), q14b (31.4 s) |
| Heavy | over 40 s | q23b (42.9 s), q4 (43.6 s) |
Those eleven take 193 seconds run one after another — 17% of the full 103-query suite. Each simulated user runs all eleven, in their own randomised order, with their own session:
| Concurrent users | Median | 95th percentile | Failures |
|---|---|---|---|
| 1 | 12.0 s | 47.1 s | 0 |
| 2 | 22.3 s | 74.6 s | 0 |
| 4 | 36.1 s | 151.7 s | 0 |
| 8 | 78.3 s | 277.9 s | 0 |
Four concurrent analysts is comfortable on this cluster shape. Eight is past the point where adding users adds queue rather than throughput.
Two things are worth knowing about how the cluster behaved under load. Individual task times stayed flat at about one second across every level — nothing was thrashing; there were simply more tasks waiting for a slot. And there were zero failures at every level, across 151,290 tasks. Under overload the cluster slows down; it does not fall over. That is a very different risk to explain to a business than one that starts erroring at peak.
This test used no think time between queries, so it is a deliberately pessimistic bound. Real analysts read their results.
What we did not test
Every benchmark has a boundary, and the useful ones say where it is.
- We did not measure concurrency at 1 TB. The concurrency figures above are at 100 GB. Both are real; neither substitutes for the other.
- We did not run Trino or Databricks ourselves. Both comparisons use published figures measured on different hardware. The only unarguable comparison is one we run end to end, and it is on our list.
- One execution per query, no warm-up. Enough for a credible total; not enough to put a confidence interval on any single query.
- No tuning of any kind. No partitioning, sort orders, compaction or caching. A production deployment would use all of them.
- These are not audited TPC results. The workload derives from the TPC-DS templates; the implementation and methodology are ours.
What this means: one engine instead of two
Most lakehouse architectures quietly run two engines. Spark builds the tables — ingestion, transformation, the nightly pipelines. Then Trino, Dremio, Presto or similar sits alongside it to serve the dashboards and the ad-hoc questions, because Spark was too slow for anyone to wait on.
That split is not free. It costs you:
- Two sets of skills. Your team learns Spark's tuning model and a second engine's. Different memory settings, different join strategies, different failure modes, different profiling tools.
- Two security and governance surfaces. Every table permission, row filter and audit requirement has to be expressed twice, and kept in step.
- Two operational footprints. Two clusters to size, patch, monitor and pay for — often both provisioned for peak, both idle at different times.
- Metadata drift. Two engines reading the same catalog, each with its own view of statistics, caching and freshness.
The whole justification for that complexity was the performance gap. When Spark is 3.5× behind a dedicated query engine, running a second one is the rational choice — the operational cost is real, and so is the reason you pay it.
At 1.49×, the arithmetic changes. The same engine that builds your tables can serve your dashboards, on the same open Iceberg tables, with one security model, one tuning model, and one thing for your team to learn. We wrote about the hidden costs of the dual-engine architecture earlier this year, and concluded then that neither engine could replace the other yet. This is the measurement that moved that line.
Be clear about the trade. You are not getting a faster engine than Trino — you are giving up about 1.5× on query wall-clock and getting back an entire system you no longer operate. If your business genuinely depends on the last 30% of query latency, a dedicated engine is still the right answer, and we would tell you so. For most teams, the second engine costs more in people and complexity than it returns in seconds.
We are not claiming to be the fastest engine on this benchmark. We are saying the gap has narrowed to the point where the second engine may no longer be worth what it costs you to run.
Run it yourself
The most useful thing we can tell you about this benchmark is how to stop trusting it.
TPC-DS is public. The data generator ships with Cazpian — the terabyte used here was created in 51 minutes with a single statement per table. The benchmark harness is part of the platform: point it at a query set, a dataset and a compute pool, and it records every execution, checks that results match across configurations, and refuses to report a query whose duration would not mean what it appears to.
If you are evaluating us against Trino, Dremio or Databricks, run all three against your own tables on hardware you control. Vendor benchmarks — including this one — are measured on data the vendor chose. Yours is the only dataset whose shape matches your business.
What we would ask is that you measure the same way we did: every query, one execution each, results checked for correctness before timing is read, and the failures published alongside the wins.
Every number in this post comes from the run's own artefacts — query times from the benchmark harness, engine metrics from Spark's history API, hardware utilisation from AWS container metrics. We are happy to walk through the methodology with anyone evaluating the platform.
Running Spark for pipelines and a second engine for analytics? Talk to our team — we can help you measure your own workload on both and work out whether the second engine is still earning its keep.
TPC-DS is a trademark of the Transaction Processing Performance Council. These are not audited TPC results and must not be presented as such. Third-party figures are quoted from the sources linked above and were measured on different hardware.