Databricks System Tables: The Observability Gap — What They Expose vs What You Actually Need for Cost Control
· 17 min read
Databricks system tables look comprehensive on paper. Sixteen tables across ten schemas. Billing, compute, jobs, queries, lineage, audit. When Databricks deprecated Overwatch and pointed everyone to system tables, the message was clear: this is the future of observability.
But if you have ever tried to answer these questions using only system tables, you already know the gap:
- Why is my job's GC time at 15%? (System tables do not track GC time.)
- Which stage is spilling to disk? (System tables do not track per-stage spill.)
- Which executor is the memory bottleneck? (System tables do not track executor-level JVM metrics.)
- How many files does my Delta table have? Is it healthy? (System tables do not track Delta table physical metrics.)
- What did my serverless job's infrastructure look like? (System tables do not populate
node_timelinefor serverless.)
This post documents exactly what Databricks system tables contain — column by column — and exactly what is missing. Every claim is verifiable against Databricks' own documentation.