Petabytes in Plain Sight: Extending Databricks to the On-Prem Data You Already Have
Overview
| Experience | In Person |
|---|
Session Speakers
Ugur Tigli
/CTO
MinIO
Full Summary
Keeping enterprise data on-prem while unlocking Databricks analytics
Enterprises want modern analytics without moving their most sensitive data out of controlled environments. A new integration between MinIO and Databricks bridges that gap by letting Databricks query on-premises data in place, at full fidelity, using open formats and Delta Sharing. The approach aligns with how the Databricks Platform governs access, avoiding copies, custom SDKs, and replication drag. For organizations evaluating whether this model fits their stack, the enterprise analytics platform evaluation guide lays out the key criteria for assessing modern data and AI platforms.
FAQ
Only the specific query results needed for the Databricks session move, and those results are ephemeral. The source tables remain on MinIO under the customer's control, the share is read-only, and sessions can be cleaned so no persistent copy remains on the Databricks side.
Data needs to be in Iceberg v3 on MinIO with a REST catalog. From there, Delta Sharing exposes the tables to Unity Catalog for direct querying. Many modern tools can land data in Iceberg format, which is the main requirement.
Masking, tokenization, and subsetting reduce data fidelity and degrade results. In-place sharing preserves full fidelity, because the original data is neither altered nor replicated, while governance and control remain anchored on-prem.
HDFS typically stores three copies of data, increasing cost at scale. MinIO uses erasure coding to remove that overhead and supports S3A, which allows existing Hadoop pipelines to connect while benefiting from lower storage footprint.
Agents generate large volumes of intermediate context and artifacts that exceed GPU memory. MinIO's work with NVIDIA on KV cache offloading, combined with a single platform spanning structured and unstructured data shared into Databricks, provides a consistent storage tier for analytics and agent applications.