Session

Petabytes in Plain Sight: Extending Databricks to the On-Prem Data You Already Have

Overview

ExperienceIn Person
Every enterprise building a lakehouse hits the same wall: the most valuable data lives on-premises. You can mask it. Tokenize it. Replicate a fraction. But data bound by HIPAA, GDPR, or sovereignty mandates cannot leave at all. Either way, Databricks never sees the full picture.Until now.MinIO and Databricks have solved this by embedding open sharing protocols directly into the data foundation. When your data foundation natively speaks open sharing protocols, data movement becomes unnecessary. Databricks queries on-prem tables directly, in place with full governance retained by the data owner.MinIO CTO Ugur Tigli shows how customer demand drove MinIO and Databricks to run analytics on full-fidelity production data on-prem through open protocols like Delta Sharing and the Apache Iceberg™ REST API. He covers what happens to your security posture when pipelines disappear, what it means when you unlock petabytes to become first-class Databricks assets.

Session Speakers

Speaker placeholderIMAGE COMING SOON

Ugur Tigli

/CTO
MinIO

Full Summary

Keeping enterprise data on-prem while unlocking Databricks analytics

Enterprises want modern analytics without moving their most sensitive data out of controlled environments. A new integration between MinIO and Databricks bridges that gap by letting Databricks query on-premises data in place, at full fidelity, using open formats and Delta Sharing. The approach aligns with how the Databricks Platform governs access, avoiding copies, custom SDKs, and replication drag. For organizations evaluating whether this model fits their stack, the enterprise analytics platform evaluation guide lays out the key criteria for assessing modern data and AI platforms.

FAQ


Only the specific query results needed for the Databricks session move, and those results are ephemeral. The source tables remain on MinIO under the customer's control, the share is read-only, and sessions can be cleaned so no persistent copy remains on the Databricks side.

Data needs to be in Iceberg v3 on MinIO with a REST catalog. From there, Delta Sharing exposes the tables to Unity Catalog for direct querying. Many modern tools can land data in Iceberg format, which is the main requirement.

Masking, tokenization, and subsetting reduce data fidelity and degrade results. In-place sharing preserves full fidelity, because the original data is neither altered nor replicated, while governance and control remain anchored on-prem.

HDFS typically stores three copies of data, increasing cost at scale. MinIO uses erasure coding to remove that overhead and supports S3A, which allows existing Hadoop pipelines to connect while benefiting from lower storage footprint.

Agents generate large volumes of intermediate context and artifacts that exceed GPU memory. MinIO's work with NVIDIA on KV cache offloading, combined with a single platform spanning structured and unstructured data shared into Databricks, provides a consistent storage tier for analytics and agent applications.