Optimize Lakehouse Cost and Performance With Intelligent Storage and Liquid Clustering
Overview
| Experience | In Person |
|---|---|
| Track | Governance & Security |
| Industry | Enterprise Technology |
| Technologies | Databricks SQL, Lakeflow, Unity Catalog |
| Skill Level | Intermediate |
Optimizing cost and performance in the lakehouse starts with how your data is stored and organized. In this session, we’ll show how Intelligent Storage and Liquid Clustering work together to automatically optimize data layout for your workloads. You’ll get a quick refresher on Liquid Clustering, then dive into what’s new, including Automatic Liquid Clustering on Unity Catalog managed tables, adaptive layouts that handle evolving query patterns, and metadata-driven optimizations that reduce unnecessary data scans.
More importantly, we’ll cover best practices for when and how to rely on automation versus manual tuning. Learn how to choose the right table types, avoid common pitfalls, and take advantage of co-clustered joins and layout-aware optimizations to accelerate queries at scale. Whether you’re managing terabytes or petabytes, walk away with practical guidance to reduce costs and improve performance without constant intervention.
Session Speakers
Jeffrey Gong
/Product Manager
Databricks
Rahul Mahadev
/Staff Software Engineer
Databricks
Full Summary
Optimizing lakehouse cost and performance with liquid clustering and intelligent storage
Modern lakehouses serve more data, more pipelines, and more users, from analysts to AI agents. Two goals remain constant, fast queries and low cost. The session shows how Unity Catalog managed tables on the Databricks Platform use intelligent storage, especially liquid clustering, to automate table layout, maintenance, and lifecycle so teams can scale without hand tuning files and partitions. It replaces much of the manual work that historically defined data engineering on the Databricks Platform, a discipline that now extends from ingestion through lifecycle management on a single unified platform covered in depth across Lakeflow Jobs and Lakeflow Declarative Pipelines.
FAQ
Yes. The speakers state liquid delivers out of the box read performance that outperforms even well tuned partitioned and z ordered tables. It also resists skew, maintains target file sizes, and adapts as data and queries change.
Minimal. Liquid conversion is GA and uses a single SQL command to replace partitions with clustering. It avoids full table rewrites, runs alongside live ingestion, and has been used with zero downtime in customer environments.
Not necessarily. Automatic liquid clustering analyzes workload telemetry and applies configurations with the highest pruning benefit. It is rolling out by default on new managed tables, with a dry run mode coming to preview recommendations.
It schedules optimize to keep data well clustered incrementally, vacuum to delete unreferenced files and cut storage costs, and analyze to refresh statistics. An AI model uses Unity Catalog telemetry to run only high value operations.
Define the expiration on a clustering column, for example date, so eligible rows are already isolated. Deletes become metadata only, and predictive optimization orchestrates the subsequent purge and vacuum steps automatically.