Session

Declarative Pipelines: What’s Next for the Apache Spark Ecosystem

Register or Login

Overview

Wednesday

June 11

4:10 pm

ExperienceIn Person
TypeBreakout
TrackData Engineering and Streaming
IndustryEnterprise Technology
TechnologiesApache Spark, DLT
Skill LevelBeginner
Duration40 min

Lakeflow Declarative Pipelines has made it dramatically easier to build production-grade Spark pipelines, using a framework that abstracts away orchestration and complexity. It’s become a go-to solution for teams who want reliable, maintainable pipelines without reinventing the wheel.But we’re just getting started. In this session, we’ll take a step back and share a broader vision for the future of Spark Declarative Pipelines — one that opens the door to a new level of openness, standardization and community momentum.We’ll cover the core concepts behind Declarative Pipelines, where the architecture is headed, and what this shift means for both existing Lakeflow users and Spark engineers building procedural code. Don’t miss this session — we’ll be sharing something new that sets the direction for what comes next.

Session Speakers

Michael Armbrust

/Distinguished Engineer
Databricks

Sandy Ryza

/Software Engineer
Databricks