2 posts tagged with "performance"

There is a long-standing assumption in data engineering: if you want fast SQL, you do not use Spark. Spark is for pipelines. For interactive analytics you bring in a dedicated query engine, run it alongside, and accept the cost of operating two systems over one set of tables.

We wanted to know how true that still is. So we ran the full TPC-DS benchmark — all 103 queries — against a terabyte of Apache Iceberg data on a four-worker Cazpian cluster, and published everything: every query time, the cost, the hardware utilisation, and the parts that are not yet good enough.

Your Spark job finished. It ran for 2 hours. It used 50 executors. The bill arrived. Now what?

Most teams have no systematic way to answer the obvious follow-up questions: Was 50 executors the right number? Did we waste memory? Was there data skew? Should we use different instance types? They either overprovision "just in case" — burning 2-5x the needed budget — or reactively debug when jobs fail with OOM errors.

The root cause is not a lack of tooling. It is a data collection problem. Spark generates extraordinarily detailed metrics at every level — task, stage, executor, application — but those metrics vanish when the application terminates. If you did not capture them during or immediately after execution, they are gone.

This post solves that problem. We cover every method for collecting Spark profiling metrics, show exactly how to set them up on Databricks, EMR, and Dataproc, and then show you how to turn those metrics into right-sizing decisions with concrete formulas and thresholds.