We Benchmarked Apache DataFusion Comet on Iceberg — 1.51x Faster on 45% Less CPU, and the Version Number That Almost Fooled Us

Every native execution engine for Spark arrives with the same headline: 2x, 3x, sometimes 5x faster. The benchmarks are usually real. What they rarely tell you is whether they describe your data, your table format, and your version matrix.

So we measured it ourselves. Apache DataFusion Comet, against 200 million rows of Apache Iceberg data, on the Spark version we ship. We recorded wall-clock time, CPU seconds, peak memory, and — the part most benchmarks skip — whether the answers came back the same.

The result: 1.51x faster on 45% less CPU, with every result identical to standard Spark. On the best query in the suite, 3.18x faster on 80% less CPU.

We also nearly published the exact opposite conclusion. Our first full round measured zero acceleration on Iceberg — not "a little", literally zero accelerated operators across five configurations. That result was real, reproducible, and completely misleading, and the reason it was wrong is worth more to you than the speedup number.

Why a native engine at all

Spark is a JVM engine. Every row it processes travels through Java objects that are allocated, tracked, and eventually garbage collected. For a lot of analytical work, that overhead exceeds the actual arithmetic.

Decimal arithmetic is the clearest case. Money columns — revenue, cost, tax, margin — are stored as decimal, because floating point cannot represent currency reliably. Spark computes decimals with BigDecimal, so every single addition inside a SUM() over 200 million rows allocates an object. The CPU spends more time in memory management than in adding numbers.

Comet attacks this directly. It is a Spark plugin that replaces individual physical operators — scans, filters, projections, aggregates — with Rust implementations operating on Apache Arrow's columnar batches. Instead of a hundred million object allocations, you get tight loops over contiguous memory.

For a lakehouse platform the appeal is not really speed. It is cost. A query that finishes in the same time on half the CPU means the same pool absorbs twice the concurrency, or a smaller pool carries the same load. That is a line item, not a leaderboard.

The benchmark, in full

We are publishing the schema and the queries, because a benchmark you cannot inspect is a claim, not a measurement.

The dataset. A synthetic sales fact table — 200 million rows, 25 columns, written as Iceberg (and separately as plain Parquet for comparison), plus a 2 million row customer dimension. Every value is derived arithmetically from the row number, so both sides of every A/B read byte-identical data.

CREATE TABLE fact_sales (
  order_id        BIGINT,
  customer_id     INT,
  region_id       INT,
  category        STRING,        -- 16 distinct
  status          STRING,        -- 5 distinct
  channel         STRING,        -- 4 distinct
  qty             INT,
  amount          DECIMAL(18,2), -- money columns are decimal,
  cost            DECIMAL(18,2), -- which turns out to matter enormously
  tax             DECIMAL(18,2),
  discount_amt    DECIMAL(18,2),
  freight         DECIMAL(18,2),
  sku             STRING,        -- 500k distinct
  order_date      DATE,
  ship_date       DATE,
  -- plus 10 filler columns most queries never touch,
  -- because that is what real warehouse tables look like
  ...
) USING iceberg;

The filler columns matter. A benchmark on a narrow table overstates nothing and understates columnar projection pruning, which is a real part of why columnar engines win.

The queries. Seven, chosen to cover the shapes a BI workload actually produces rather than the shapes that flatter a vectorized engine. Three representative ones:

-- Decimal arithmetic over a large scan. This is the one Comet transforms.
SELECT region_id,
       sum(amount * qty)                                        AS gross,
       sum((amount - discount_amt) * qty - cost - freight)      AS net,
       avg(amount)                                              AS avg_ticket
FROM fact_sales
WHERE status <> 'CANCELLED'
GROUP BY region_id ORDER BY region_id;

-- A normal BI report: date range, filters, decimal aggregates, modest group-by.
SELECT category, channel, count(*) AS orders,
       sum(amount)                          AS revenue,
       sum(amount - cost - discount_amt)    AS margin
FROM fact_sales
WHERE order_date >= DATE '2024-01-01'
  AND status IN ('SHIPPED','DELIVERED')
  AND amount > 100.00
GROUP BY category, channel ORDER BY category, channel;

-- Wide table, narrow projection: 5 of 25 columns, selective predicate.
SELECT channel, count(*) AS n, sum(amount) AS revenue
FROM fact_sales
WHERE order_date BETWEEN DATE '2024-03-01' AND DATE '2024-09-30'
  AND attr_i3 < 20
GROUP BY channel ORDER BY channel;

The other four: a pure filtered count, a global decimal aggregate, a set of string predicates, and a join to the customer dimension.

Four precautions, because benchmarks are easy to bend:

• Equal memory budgets. Comet needs off-heap memory for Arrow buffers, so it is trivial to hand it more RAM and call the difference a win. Both sides got exactly 14 GB — Spark as 14 GB heap, Comet as 10 GB heap plus 4 GB off-heap.
• Correctness before speed. Every query emits a checksum over its complete result set in both modes. A configuration that returns different answers is not faster, it is broken.
• CPU measured at the container, not in Spark. Comet executes in its own native thread pool, which Spark's task metrics never attribute to a task. Measuring executorCpuTime systematically undercounts the accelerated side. We read total container CPU from the cgroup — which is also what cloud providers bill.
• Cold and warm separated. First execution includes planning and native runtime initialisation; we report it apart rather than averaging it in.

The result on Iceberg

Query	Spark	Comet	Faster	CPU saved
Filtered count	6.19s	4.83s	1.28x	41%
Decimal global aggregate	9.96s	7.10s	1.40x	46%
Decimal margin by region	28.73s	9.04s	3.18x	80%
Narrow projection, wide table	3.08s	2.45s	1.26x	50%
String predicates	3.92s	3.42s	1.15x	32%
BI report	7.76s	6.30s	1.23x	38%
Join + aggregate	25.30s	23.24s	1.09x	0%
Total	84.9s	56.4s	1.51x	45.0%

All seven byte-identical.

The standout is the decimal margin query: 28.7 seconds and 298 CPU-seconds became 9.0 seconds and 59. That single query is the entire argument for native execution — almost pure decimal arithmetic over a large scan, where Spark spends nearly all its time allocating BigDecimal objects.

The join barely moved. Joins are shuffle-dominated, and shuffle is the part we deliberately left on Spark (more on that below).

The same suite on plain Parquet

We ran the identical seven queries against the identical data written as plain Parquet, to separate "what does the engine do" from "what does the table format cost".

Query	Spark	Comet	Faster	CPU saved
Filtered count	8.56s	5.86s	1.46x	62%
Decimal global aggregate	15.47s	11.37s	1.36x	58%
Decimal margin by region	29.22s	10.37s	2.82x	86%
Narrow projection, wide table	4.13s	3.25s	1.27x	35%
String predicates	6.25s	5.03s	1.24x	37%
BI report	11.36s	8.19s	1.39x	53%
Join + aggregate	12.40s	14.55s	0.85x	−115%
Total	87.4s	58.6s	1.49x	44.5%

1.49x and 44.5% on Parquet against 1.51x and 45.0% on Iceberg. Those are the same number twice, which is the point: the table format was never the limiter. Once the engine is actually running, Iceberg costs you nothing relative to raw Parquet — the opposite of what our first round appeared to show.

Two details from this run worth carrying over.

Peak memory fell from 10.76 GiB to 3.62 GiB. Comet processes columnar batches off-heap instead of materialising rows as JVM objects, and it shows. Treat that number with a caveat — the two sides were configured with different heap sizes (14 GB versus 10 GB) to hold the total budget equal, so some of the drop is the JVM simply having less room to grow into. But 3.6 GiB peak against a 10 GB heap ceiling means Comet never needed the headroom, which is the part that matters for sizing.

The join got materially worse here: 0.85x, on more than double the CPU. This was not a fallback — the plan was almost entirely native. Comet's broadcast hash join, building a hash table over a 2 million row dimension, is simply more expensive than Spark's for this shape. On Iceberg the same query was neutral (1.09x). If your workload is dominated by broadcast joins over large dimensions, measure before you enable — this is the one shape where we saw a real regression, and we would rather you hear it from us.

The schema lesson

Our very first round used double for the money columns and produced only 1.20x. We nearly filed that as Comet's ceiling. It was our schema.

Money column type	Result	Why
double	1.20x	strictFloatingPoint sends every float aggregate back to Spark
decimal(18,2)	1.49x	Aggregates run natively, and decimals are Spark's weak spot

Comet has a safety setting, spark.comet.exec.strictFloatingPoint, which refuses to run floating-point aggregates natively because native summation accumulates in a different order and would change the last digits. With double columns that setting left only the scan accelerated. Switching to decimal — which is what money should have been stored as anyway — unblocked the aggregates.

We verified the determinism concern is real rather than theoretical. On double columns the accelerated and standard sums differed by about 1.9×10⁻¹³ relative, roughly three units in the last place, from ordinary floating-point non-associativity. Integer aggregates matched exactly, and every decimal aggregate matched exactly. If you run differential testing across engines, define a relative tolerance for float columns before you start, or you will generate false failures on every SUM() of a double.

The version number that almost fooled us

Our first round measured Iceberg at 1.04x with zero accelerated operators, across five configurations including the two that exist specifically to enable Iceberg acceleration. We had a well-evidenced, reproducible, wrong conclusion — and we were close to publishing it.

The cause was a single version mismatch. Comet's native Iceberg reader reflects into Iceberg's internals to extract file scan tasks. The version we were testing had been validated against Iceberg 1.5 through 1.10. We were on Iceberg 1.11.0 — and reflection failed:

Comet cannot accelerate BatchScanExec because:
  Failed to extract Iceberg metadata via reflection

There is no way to configure around that, and no older Iceberg to fall back to: Iceberg publishes exactly one Spark 4.1 runtime, at 1.11.0. On Spark 4.1, that older Comet had no working Iceberg combination at all. Upgrading Comet fixed it completely.

Two lessons we would pass on. First, when a native engine reports no acceleration, check the version matrix before you conclude anything about the technology — reflection-based integrations are exactly the kind that break quietly on a minor version bump. Second, and more useful: 0 in an operator count is a diagnostic, not a performance result. Had we only measured wall-clock time, 1.04x would have looked like "native execution just does not help us much here", and we would have drawn a sweeping and false conclusion from it. The operator count is what told us the engine was not running at all.

Two ways to silently get nothing

Related, and worth knowing before you enable this anywhere. In current Comet, the extension disables itself entirely unless you make an explicit choice about shuffle:

Comet extension is disabled because spark.shuffle.manager is not set to
org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager.
Set spark.comet.shuffle.enabled=false to keep Comet enabled with
Spark's default shuffle manager.

Set one or the other. If you set neither — which is easy, because neither was required in earlier versions — you get a plugin that loads, reports itself enabled, and accelerates nothing. That is a worse failure than crashing, because it is invisible.

We chose spark.comet.shuffle.enabled=false, keeping Spark's own shuffle. That is the conservative option, and it turned out to also be the more robust one: with Comet's native shuffle enabled, our join query died with an off-heap allocation failure inside Comet's native sorter, while the same query completed normally on Spark's shuffle. The numbers above are all from the conservative configuration — which means they are a floor, not a ceiling.

What this means

Native acceleration on Iceberg is real, and the cost story is the interesting one. A 45% reduction in CPU at identical results is not a micro-optimisation. On per-second cloud billing that is close to halving the compute cost of an analytical workload.

Your schema decides how much you get. decimal money columns gave 1.49x where double gave 1.20x, for the reason above. If you benchmark a native engine and are underwhelmed, check your column types before you check anything else.

Iceberg costs you nothing here. 1.51x on Iceberg against 1.49x on raw Parquet. Choosing an open table format does not mean giving up native execution.

Shuffle-heavy work benefits least, and joins can regress. Our join gained 1.09x on Iceberg and lost 15% on Parquet while burning double the CPU. If your workload is mostly large joins, temper expectations and measure; if it is scans, filters and decimal aggregates, the headline numbers are representative.

How we are shipping it

Native acceleration appears in Cazpian as a single control on the compute pool, with three settings: Automatic, Off, and Required. Automatic accelerates when the runtime and pool size support it and quietly runs standard Spark when they do not, so you never get a pool that fails to start because acceleration was unavailable. Required is for administrators who would rather a pool refuse to start than run unaccelerated — the right choice for a canary or a benchmark.

Acceleration is opt-in per approved runtime image, and eligibility is bounded by a validated memory envelope rather than a percentage of container memory. That is deliberate: an engine that needs off-heap memory can quietly overcommit a container and turn a performance feature into an outage, and our own native-sorter failure above is exactly that risk showing its face.

We will keep publishing these numbers as the versions move — including the rounds where the answer is "no change" or "slower". A benchmark that only ever produces wins is not a benchmark.

---

Measurements: Spark 4.1.0, Scala 2.13, Iceberg 1.11.0, 200M rows across 25 columns, 15 CPUs, equal 14 GB memory budget per side, p50 of four repetitions with the cold run reported separately. Result equivalence verified by full result-set checksum on every query. Comet configured with Spark's default shuffle manager.