100 Concurrent Spark Jobs, 67 Seconds, One $2/Hour Pool
We submitted one hundred Spark jobs at once, each scanning ~200 MB of Iceberg data. All of them finished in 67.5 seconds on a single small compute pool — and we kept pushing until 500. Here are the measured numbers, what broke along the way, and what it means for your smallest, most expensive jobs.
Read the post →
We Moved Cazpian's Spark Pools to ARM — 2x Faster Under Load
An honest A/B of Cazpian Spark compute pools on AWS Graviton (ARM64) versus x86: 400/400 jobs succeeded, ARM ran each job roughly twice as fast under 100-way concurrency at ~20% lower Fargate cost — and the 8.5x 'ARM regression' we found first turned out to be an infrastructure outage wearing three disguises. Here is the data, the wrong turns, and the checklist we wish we'd had.
What a Terabyte Actually Costs: TPC-DS on Cazpian
We ran all 103 TPC-DS queries against a terabyte of Iceberg data on four workers. Every query passed. Here are the times, the cost, the resource utilisation, and the parts we are still working on.
We Benchmarked Native Query Acceleration on Iceberg — Up to 4.14x, and the Version Number That Almost Fooled Us
Native execution promises big speedups on Spark. We measured Cazpian Native Query Acceleration against 200 million rows of Iceberg data, reporting CPU cost and result correctness alongside wall time — 1.51x across a mixed suite, 4.14x on decimal aggregates through the catalog — and nearly published the wrong conclusion because of a single version mismatch.
The Serverless Black Box: What You Lose on Databricks Serverless Compute
Databricks serverless compute removes infrastructure management but also removes your ability to observe, tune, and control costs. Here is exactly what you give up — from blocked Spark configs to missing metrics to 2-3x higher prices.
Databricks System Tables: The Observability Gap — What They Expose vs What You Actually Need for Cost Control
A detailed analysis of Databricks system tables — the exact column schemas of node_timeline, billing.usage, lakeflow, and query.history — and the critical metrics missing for cost control: no executor GC time, no per-task shuffle, no disk spill per stage, no peak memory per executor, no Delta table health, and no per-query cost attribution. Includes the serverless gap, the Overwatch deprecation problem, and the Iceberg metadata comparison.
How to Profile Spark Jobs After Completion: The Complete Guide to Collecting Metrics on Databricks, EMR, and Dataproc
The complete guide to collecting and analyzing Spark job profiling metrics after completion. Covers Spark event logs, History Server REST API, Prometheus integration, SparkListener, and platform-specific collection on Databricks (system tables, Overwatch), AWS EMR (CloudWatch, persistent History Server), and GCP Dataproc — with right-sizing formulas and cost optimization thresholds.
Iceberg Scan and Commit Fine-Tuning: The Production Operations Guide for Spark
The complete production operations guide for Apache Iceberg on Spark — when to use partition, sort order, or bloom filter (and when to combine them), how to minimize compute by skipping unnecessary file reads, small file and manifest compaction, snapshot hygiene and metadata file control, why Iceberg v2 is essential, delete file management, and the full maintenance lifecycle with recommended thresholds.
Spark Runtime Metrics Collection with DriverPlugin, ExecutorPlugin, and SparkListener
A complete guide to collecting Apache Spark runtime metrics using DriverPlugin, ExecutorPlugin, SparkListener, and QueryExecutionListener — covering compute metrics, Iceberg scan and commit metrics, unified metrics pipelines, scalable storage alternatives to PostgreSQL (TimescaleDB, ClickHouse, VictoriaMetrics, Iceberg tables on S3), production architecture patterns, and a full Java implementation walkthrough.
The Complete Apache Spark and Iceberg Performance Tuning Checklist
The definitive performance tuning checklist for Apache Spark and Apache Iceberg — covering cluster sizing and executor memory, Kryo serialization, shuffle optimization, join strategies and broadcast thresholds, Adaptive Query Execution (AQE) configuration, predicate pushdown and column pruning, caching strategy, data skew fixes, Parquet file format and compression codec selection, Iceberg table design and partitioning, partition pruning and bloom filters, MERGE INTO and CDC write optimization, compaction and table maintenance, AWS S3FileIO tuning, GC tuning with G1GC and ZGC, Spark UI monitoring, and a complete configuration reference with recommended production values.
Spark Execution Plan Deep Dive: Reading EXPLAIN Like a Pro
A complete guide to reading Apache Spark execution plans — the Catalyst optimizer pipeline from parsed logical plan through analyzed, optimized, and physical plans, every EXPLAIN mode (simple, extended, formatted, codegen, cost), physical plan operators including FileScan, Exchange, Sort, joins, and aggregations, whole-stage code generation and the asterisk notation, predicate pushdown verification with PartitionFilters vs PushedFilters vs DataFilters, Adaptive Query Execution runtime rewrites, Spark UI correlation, anti-pattern detection, and a full annotated query walkthrough.
Spark OOM Debugging: The Complete Guide to Fixing Out of Memory Errors
A comprehensive guide to debugging every type of Spark OutOfMemoryError — Java heap space, GC overhead limit exceeded, container killed, SparkOutOfMemoryError, driver vs executor OOM — with step-by-step debugging workflows, GC tuning for G1GC and ZGC, Prometheus and Grafana observability, memory anti-patterns, and how Cazpian eliminates the guesswork.