How to Build Agentic Pipelines with OSS Spark Declarative Pipelines

Apache Spark's pipeline story has evolved dramatically— from DStreams to Spark Structured Streaming and now Lakeflow Spark Declarative Pipelines— each generation simplified how we reason about data. But with the rise of agentic AI and vibe coding, how do we ensure our most critical data pipelines remain deterministic, testable, and production-ready? In this session, Scott Haines (Staff Developer Advocate) and Allison Wang (Staff Software Engineer, Databricks and Spark Contributor) will dive into Spark's new Declarative Pipelines API, exploring how to harness the productivity gains of agentic workflows without sacrificing the reliability your data demands. You'll leave with practical patterns for building pipelines that are both developer-friendly and production-hardened and last but not least— open source.