SESSION

Data Productivity at Scale

Accept Cookies to Play Video

OVERVIEW

EXPERIENCEIn Person
TYPELightning Talk
TRACKData Engineering and Streaming
TECHNOLOGIESDeveloper Experience, ETL, SQL Analytics / BI / Visualizations
SKILL LEVELIntermediate
DURATION20 min
DOWNLOAD SESSION SLIDES

When it comes to developing new data pipelines and evolving existing ones, the available tooling lags behind what's available to software developers. Creating new development environments is cumbersome, populating these environments with representative data is compute-intensive, and the process of deploying changes to production is ad-hoc and error-prone. All of these factors drive up development costs, slow down iteration speed, and make data less reliable.

 

At the core of the open-source project, SQLMesh is a novel development workflow born from our collective experience at companies like Airbnb, Apple, Google, and Netflix. It is designed to handle the complexities of evolving data pipelines at an internet scale. In this talk, we'll discuss challenges faced by data practitioners today and highlight core SQLMesh concepts addressing them, including virtual data environments, automatic data contracts, and accurate change previews.

SESSION SPEAKERS

Iaroslav Zeigerman

/Co-Founder & Chief Architect
Tobiko Data