Session

Magnite's Journey of Interoperability on a Petabyte of Iceberg Data

Overview

ExperienceIn Person
TrackGovernance & Security
IndustryEnterprise Technology
TechnologiesUnity Catalog
Skill LevelIntermediate

For Magnite's business, they have a need to handle the ingest of 900TB of data every 10 days, while offering subsets of that data to many different customers (internal and external). Join this session to learn how Magnite successfully accomplished this while centralizing their data access via Unity Catalog while improving performance and handling many different engines (including Apache Spark, DuckDB, and Snowflake). We'll discuss the different approaches required, best practices followed, and rough edges encountered along the way. This lively presentation will be beneficial to anyone looking to separate compute and storage in their data domain at any scale.

Session Speakers

Brian Likosar

/Sr. Specialist Solution Architect
Databricks

Kayvon Raphael

/Head of Data Engineering
Magnite

Full Summary

Inside Magnite's petabyte-scale migration to Apache Iceberg

At ad tech scale, table format is a business decision. Magnite processes roughly 2 trillion ad requests per day and persists about 400 terabytes of data daily across a hybrid estate of on‑prem and cloud systems. The conversation examines why Magnite moved its core shared datasets to Apache Iceberg on the Databricks Platform, what improved, and where tradeoffs remain. It also challenges the idea that every team needs Iceberg now.

FAQ


Iceberg shines when you need true interoperability across multiple engines or clouds, direct data sharing with external partners, and data science access to raw data without copies. If your data lives cleanly inside one engine today, the benefits are weaker.

High‑frequency streaming writes performed better with Delta Lake's log‑based metadata. Iceberg's per‑commit snapshot and manifest rewrites made it a better fit for gold tables consumed by many downstream users, not for streaming intermediates.

Magnite found the automated background compaction and reclustering ran at reasonable cost, replacing manual tuning at petabyte scale. It improved file and layout health without table‑specific babysitting.

Iceberg's catalog spec covers catalog‑level permissions. There is no shared, cross‑vendor standard that enforces fine‑grained access in the object store. Implementations like Unity Catalog provide those controls.

Copy Into is simple for small tables but struggled at Magnite's volumes. Spark jobs that read from S3 into a DataFrame, normalize, and write directly to the table were faster and cheaper at scale.