2 posts tagged with "vendor-lock-in"

Databricks serverless compute promises a simple deal: stop managing clusters and just run your workloads. No instance selection. No autoscaling policies. No driver sizing. Just submit your query or job and let Databricks handle the rest.

The pitch is compelling. The reality is a black box that removes not just infrastructure management, but your ability to observe what is happening, tune how it runs, and control what it costs.

This is Part 3 of our Databricks observability series. In the previous post, we documented how system tables leave critical metrics gaps. Serverless makes those gaps dramatically worse — because on serverless, you lose even the tools that classic compute provides.

"Which platform is cheapest for Spark?" is one of the most common questions data teams ask — and one of the most misleading. The honest answer is: it depends entirely on your workload shape.

A platform that saves you thousands on large nightly batch jobs might quietly waste thousands on your fleet of small ETL runs. The billing model that looks transparent at first glance might hide costs in cold starts, minimum increments, or idle compute you never asked for.

In this post — Part 3 of our compute cost series — we compare Databricks, Amazon EMR, and Cazpian across three realistic workload scenarios. No hypotheticals. Real pricing. Real math.