13 Billion Rows into Apache Iceberg in 7 Minutes, for $3.69
A post making the rounds on LinkedIn showed a serverless Spark platform processing "1 TB (40 billion rows)" in 14 minutes 5 seconds: 91 raw Parquet files ingested into a table clustered by date and warehouse. Read closely, the 40 billion is rows read — the post notes the engine read three times more rows than it wrote, 13.3 billion, because clustering on write re-processed the data. It is a good benchmark all the same: no toy data, no caching tricks, a real shuffle and a real write, and a table you can query the moment the job commits.
So we ran the same shape of job on Cazpian.
12,959,823,744 rows — 1.13 TB of raw Parquet — landed in an Apache Iceberg table partitioned by day and sorted within each day, in a median of 7 minutes 4 seconds on 32 AWS Graviton workers. Every row counted, in three runs out of three. About $3.69 of compute per load. Starting from nothing — no cluster running at all — clicking Create Pool to the committed table took 12 minutes 3 seconds.
