Decathlon’s Race to Unified Governance: Securing Interoperability at Scale
Overview
| Experience | In Person |
|---|---|
| Track | Governance & Security |
| Industry | Retail & Consumer Goods |
| Technologies | Unity Catalog |
| Skill Level | Intermediate |
| DOWNLOAD SESSION SLIDES | |
Decathlon, operating in 70+ countries, aims to make wellbeing accessible through sport. Scaling to thousands of tables strained our legacy metadata management (AWS Glue). Moving to Unity Catalog (UC) unified governance for 2,000+ users but created a key challenge: How to centralize governance within the Databricks Lakehouse while maintaining interoperability with external engines?
This session details Decathlon’s migration from AWS Glue to Unity Catalog, focusing on how we ensured "openness" so that external engines—EMR, Athena, Polars, DuckDB—remain compatible and performant under UC. We will share the framework used to automate the transition of thousands of tables and present metrics on improved metadata rendering speeds and streamlined platform administration.
Session Speakers
Benjamin Laot
/Data Platform Staff Engineer
Decathlon
William Conti
/Staff FDE
Databricks
Full Summary
Inside Decathlon's Unity Catalog migration: the real cost of interoperability
Decathlon's move from AWS Glue to Unity Catalog offers a pragmatic view of what it takes to unify governance while keeping select non-Databricks engines in play. The team began with an ambition of full interoperability across existing compute, then confronted the integration tax of making every engine work cleanly with Unity Catalog at scale. For teams evaluating how to approach data governance on the Databricks Platform, Decathlon's experience surfaces the real trade-offs between engine flexibility and operational overhead.
FAQ
External tables avoid moving underlying data and act as pointers. Consumers can immediately repoint pipelines to Unity Catalog, while producers shift write workloads at their own pace. Managed tables and decommissioning Glue become a later, cleaner step.
Catalog commit enables external engines to write to Unity Catalog managed tables while respecting fine-grained access controls and multi-statement transactions. It is activated by setting the Delta Lake table property for catalog-managed commits and requires recent Databricks and external engine runtimes.
At the time, Decathlon needed very fast cluster spin-up and cost-effective shared workloads that serverless did not yet match. Polars on a mutualized EKS cluster filled that gap, so the team accepted the integration cost for that specific use case.
Polars cannot create tables directly, the official Unity Catalog extension is read-only, deletion vectors are unsupported by the Rust Delta Lake implementation, lineage is not captured, and row filters or column masks on external tables block access.
Not yet. With catalog commit, reads and inserts work, but updates and deletes are not supported in the demonstrated path. Teams should design for insert-oriented patterns until support expands.