SESSION

Effective Lakehouse Streaming with Delta Lake and Friends

OVERVIEW

EXPERIENCEIn Person
TYPEBreakout
TRACKData Engineering and Streaming
INDUSTRYEnterprise Technology, Retail and CPG - Food
TECHNOLOGIESApache Spark, Delta Lake, Orchestration
SKILL LEVELIntermediate
DURATION40

In life, we learn to crawl before we walk and run. This is the natural progression as we train ourselves to operate at different speeds, maintain balance, and learn to listen to and monitor ourselves and our surroundings. The same can be said for any streaming data system, especially for the lakehouse. In this session, we will discover the true power of the streaming lakehouse architecture, how to achieve success at scale, and, more importantly, why Delta Lake is the key to unlocking a consistent data foundation and empowering a "stress-free" data ecosystem. Key lessons:

 

  •  Learn practical industry standards for simplifying Data Capture and Ingestion using protocol streams.
  •  Fall in love with Schema Enforcement and Evolution.
  • Dive into the Medallion Architecture and learn why Data Quality matters.
  • Discover how Internal and External Data Domains insulate Data Teams and support the Streaming Foundation.
  • Unlock Reliable Streaming with Opinionated Workflow Orchestration.

SESSION SPEAKERS

Scott Haines

/Distinguished Software Engineer
Nike