One Write Path, Multiple Entries: How Rippling uses Unity Catalog-managed Apache Iceberg™

This is a guest post from Tae Lee, Staff Engineer, Data Platform at Rippling

At Rippling, the engine that writes a table is not always the engine that reads it. A Databricks job might produce a table, but the downstream consumer may be Rippling Data Cloud's Trino-based query layer, Snowflake, or another lakehouse reader. That makes the catalog decision more than a metadata detail: it determines who can write the table, who maintains it, and how other engines read it.

That means the goal is not to force every workload into one engine or one catalog. The goal is to let each producer use the platform where it runs best, while still publishing governed tables that downstream systems can consume without duplication.

Unity Catalog-managed Iceberg tables are the pattern we use for the Databricks-produced slice of that architecture.

Catalog Ownership Follows The Writer

Our catalog strategy starts with a simple rule: the catalog should follow the writer.

For AWS-native compute, we use AWS Glue Data Catalog. That is the natural fit for workloads written by AWS engines such as Glue, Athena, EMR, or related infrastructure.

For Databricks-produced workloads, we use Unity Catalog. This is especially important for Apache Iceberg™. Databricks can read foreign Iceberg tables through federation, but those tables are not the same as Unity Catalog managed Iceberg tables. If Databricks needs to write the table, or if the workload heavily depends on Databricks performance and governance, the table should be UC-managed.

For Snowflake-produced data that needs to be shared across platforms, we use Snowflake Horizon Catalog with Snowflake-managed Iceberg tables. That is a separate catalog decision driven by Snowflake as the producer.

This is the important distinction: Iceberg gives us an open table format, but the catalog still owns metadata commits, permissions, and table lifecycle. Format portability and catalog ownership are related, but they are not the same thing.

Why Managed Iceberg For Databricks Workloads

Before this pattern, a Databricks-produced output that needed to be consumed elsewhere usually meant an extra handoff: export a second copy after the Databricks write, or write directly into the consuming system and read it back through a connector or federation path when Databricks needed it again. Both approaches work, but they add duplicate materialization, extra compute, and connector-specific limits. 

Unity Catalog managed Iceberg gives us a cleaner boundary.

Databricks writes the table natively. Unity Catalog owns the table metadata, access model, and lifecycle. The table data and Iceberg metadata are stored in Rippling-owned S3 storage. 

External engines connect through the Iceberg REST Catalog API or catalog federation. Credential vending gives those engines scoped access to the underlying storage. The engines then read the Parquet files directly from S3 using their own compute. 

That is the key property for us. A Databricks job can produce the table once, and a downstream engine such as Trino, Snowflake, Spark, Athena, or EMR can consume the same table through standard Iceberg access patterns. The consumer does not have to proxy every query through another engine's SQL dialect or execution layer, which reduces dependence on cross-engine translation, pushdown behavior, throttling, and producer-side compute for downstream reads. 

It is not a Databricks export. It is an open Iceberg table with a Databricks-native producer.

From ML Outputs To Data Cloud

The pattern is most useful when a Databricks-produced output needs to become part of a broader product or analytics surface.

At Rippling, ML and AI workloads still choose the target that fits the use case: Delta tables, vector databases, OpenSearch, or other purpose-built destinations. Managed Iceberg is a selected publication pattern, not the default target for every ML output.

Where managed Iceberg matters is the handoff. For selected outputs, a Databricks job can publish a Unity Catalog-managed Iceberg table once. Rippling Data Cloud can then consume that same table through its query layer, including Trino-based access patterns, and use it in downstream transformations, dashboards, and AI-driven product features.

The architecture looks like this:

That gives us one governed table and one write path. Downstream consumers do not need a duplicate physical copy, and Lakeflow Jobs do not need to write separately into every consuming system.

Operations Matter As Much As Openness

Open table formats are only part of the story. Iceberg tables still need maintenance: file compaction, snapshot expiration, orphan cleanup, and statistics.

For Glue-cataloged Iceberg tables, that maintenance belongs to the AWS-native platform path. Glue has table optimization features, but it is still a separate operational model: we need to decide where to enable those features, how to monitor them, and how to validate behavior for the workloads that use Glue.

For Unity Catalog-managed Iceberg, Databricks handles more of that lifecycle through Predictive Optimization, including automatic table maintenance, file optimization and compaction, statistics collection, and data layout optimization for managed tables. That is useful because the same platform that writes the table also manages much of the hygiene required to keep it performant.

This is one reason we do not view UC-managed Iceberg as only an interoperability feature. It is also an operational model. For Databricks-produced tables, the maintenance path matters as much as the read path.

Being Precise About Lock-In

This architecture reduces data lock-in, but it does not eliminate all dependencies.

The data lock-in is low. The table is Iceberg over Parquet in Rippling-owned S3 storage.

Downstream engines can read the data directly through open Iceberg patterns.

The catalog and governance dependency is real. Unity Catalog remains the control plane for metadata, permissions, lineage, and managed table behavior. If we moved away from Databricks, the data would be portable, but we would need to replace the catalog and maintenance system.

That is an acceptable trade-off for this class of tables. UC earns its place when Databricks is the producer and the table needs to be governed, maintained, and readable by other engines.

The Practical Outcome

The value of Unity Catalog managed Iceberg is not that it gives us one catalog for every table. It does not, and that is not our goal.

The value is that it gives us one write path for Databricks-produced tables that need open downstream consumption. Databricks gets the native write and optimization path. Downstream systems get direct access to open data in S3. 

Engines that access the table through the REST catalog go through the same metadata and access layer rather than bypassing governance

through raw storage paths. That is the practical outcome:

"Unity Catalog and managed Iceberg give us the best of both worlds: native performance for our AI and ML pipelines, and open interoperability for every downstream consumer. One write path, zero duplication, and a governance layer every engine respects, including the AI-driven products we're building for Rippling's Data Cloud."

For Rippling, interoperability is not about making every engine interchangeable. It is about letting each engine do the job it is good at while keeping the published table portable, governed, and usable by the systems that need it.

To learn more about Unity Catalog and Iceberg support, visit the Unity Catalog product page.