Spark Declarative Pipelines Overview

Michael Armbrust, Distinguished Engineer at Databricks, explains why declarative programming is important in data engineering. He reviews an example use case and walks through some common challenges. In the end, he demonstrates how you can simplify building and operating data pipelines with Lakeflow.

Spark Declarative Pipelines helps you simplify batch and streaming ETL with automated reliability and built-in data quality. Part of Lakeflow.

Watch this video to learn more about Spark Declarative Pipelines for efficient ingestion, intelligent data transformation, and automated operations.