Session

Interoperability With Unity Catalog: Enabling Governed Data Access Across Platforms Beyond Databricks

Overview

Experience	In Person
Track	Governance & Security
Industry	Enterprise Technology, Healthcare & Life Sciences, Financial Services
Technologies	Databricks SQL, Unity Catalog, Genie
Skill Level	Advanced

Many enterprises run their data across multiple platforms — Snowflake, BigQuery, OneLake, AWS Glue, Postgres. In 20 minutes we'll show you how Lakehouse Federation makes Unity Catalog the catalog of catalogs for your entire data estate.

You'll see demos on how to connect an external platform to UC, then run Genie across federated data with row-level security applied uniformly across every source.

Session Speakers

Liran Bareket

/Lead Product Specialist (Lakebase)
Databricks

IMAGE COMING SOON

Michael Saltzman

/Specialist Solutions Architect
Databricks

Full Summary

Govern every platform from Unity Catalog: the catalog of catalogs approach

Most organizations run multiple data platforms in parallel, and governance rarely waits for migration. The session proposes treating Unity Catalog as a catalog of catalogs: federate data where it sits, apply one consistent policy, and migrate only when performance or feature needs demand it.

Unity Catalog on the Databricks Platform gives teams governed access to data across Snowflake, BigQuery, Redshift, Postgres, AWS Glue, Salesforce, and more through Unity Catalog on the Databricks Platform. Teams looking to understand how governance fits into a broader strategy will find the data governance approach to unified data analytics a useful foundation before exploring federation in depth.

The approach unifies access controls, row and column security, and lineage across both lakehouse and federated sources. Interoperability works in both directions and an incremental path upgrades assets from foreign to fully managed tables, unlocking performance and operational advantages without moving data out of a customer's own storage.

One policy across every catalog through federation

Unity Catalog sits between clients and external catalogs so any Databricks or compatible client can reach federated sources under a single policy model. The same access controls, row filters, column masks, and lineage that govern lakehouse tables apply to data in other systems.

Federation is bidirectional. Unity Catalog can read from external catalogs, and external catalogs can read Unity Catalog tables, including Iceberg. The reading system enforces governance. The recommended pattern is to write in one catalog and read from the other, keeping ownership clear and avoiding dual writes, which are possible but discouraged.

Incremental path: foreign to external to managed

Teams can graduate assets in place, moving from read-only foreign tables to fully managed Databricks tables without ETL. The sequence is straightforward: use ALTER TABLE SET EXTERNAL to convert a foreign table to a read and write external table (GA), run CONVERT TO DELTA to upgrade Parquet to Delta Lake where applicable, then use ALTER TABLE SET MANAGED to promote it to a managed table while data remains in the customer's S3 bucket.

Managed tables deliver tangible benefits that reduce cost and improve reliability:

Liquid clustering and predictive optimization tuned to query patterns
Automatic compaction and storage cleanup on delete
Undrop windows for recovery
Catalog-managed commits that prevent storage contention
Multi-statement transactions for atomic changes across multiple tables
Stronger governance by eliminating metadata drift and unauthorized paths

Natural language analytics without migration

Analysts can start asking questions immediately across federated and native Unity Catalog data. A Genie-based example showed natural language queries and visualizations working across Glue and Databricks-managed tables together. Business users can join and analyze data living in systems like Redshift, Snowflake, or Glue alongside lakehouse data, without waiting for an ETL project.

Roadmap for open, governed interoperability

The product roadmap strengthens open connectivity while preserving centralized governance. Recently delivered and in-preview items include:

Custom JDBC connections (preview), Palantir federation (preview), remote query (preview), set managed (preview), JDBC join pushdowns (preview), OAuth for Snowflake (GA)
Iceberg V3 federation (GA)
Lakehouse Federation sharing of foreign tables via Delta Sharing, using materialization so consumers do not access source systems directly
Federated comments (preview), Iceberg partition evolution (preview)
IAM role authentication for Redshift, Postgres, and MySQL
Additional IRC federation, including BigQuery

Coming next:

Schema filtering
A dedicated convert-foreign-to-managed command for Iceberg (available today via CTAS, to be simplified)
Foreign table disaster recovery and automated refresh
More custom JDBC foreign catalogs and IRC endpoints

Policy-first governance for a multi-platform reality

Federate first, govern in Unity Catalog, then migrate when ready. That sequence imposes one set of policies and lineage across a complex estate today, while creating a clean on-ramp to the performance and operational benefits of managed tables as teams standardize over time.

FAQ

No. Federation lets Unity Catalog apply access controls, row filters, column masks, and lineage to data that stays in its source system. Migration becomes an optional step driven by performance or feature needs.

Foreign tables are read-only references to data in another catalog. ALTER TABLE SET EXTERNAL converts a foreign table to a read and write external table that remains in your storage. ALTER TABLE SET MANAGED promotes it to a Databricks-managed table, enabling optimizations and stronger governance.

Managed tables add liquid clustering, predictive optimization, automatic compaction, undrop, catalog-managed commits, and multi-statement transactions. Teams gain lower cost, better performance, and tighter governance by preventing metadata drift and unauthorized access paths.

Yes. Lakehouse Federation sharing allows sharing of foreign tables with downstream consumers without granting access to the underlying source system, using materialization under the hood.

It is technically possible but discouraged. The recommended pattern is to write in one catalog and read from the other, with the reading system enforcing governance.