Lakeflow: Unified agentic data engineering

A data pipeline usually refers to a multistep process of moving and processing data. Data pipelines are key to data engineering and are used to ingest data from a data source, clean it, modify it, enrich it and make it usable. Therefore, data pipelines are part of almost every analytics or AI initiative. Data pipelines can be simple or complex and often need to handle different types of data (e.g., structured or unstructured data) and data in real time, and also have to be able to scale to process large amounts of data.

Data streaming refers to the data that is generated in real time and is often organized and consumed through data streams. Today, more and more data is generated in real time by sensors, websites, mobile applications, etc. Organizations that are looking to use data streams for implementing solutions such as real-time analytics often find that data streams can be challenging to manage and process efficiently. This is why many Databricks customers turn to building on the Data + AI Platform, which offers built-in streaming capabilities that make stream processing easy, efficient and highly scalable.

Databricks provides a powerful, unified data engineering engine through Lakeflow, designed to handle everything from ingestion to transformation to orchestration at scale. Core capabilities include ingesting data from a wide range of sources with Lakeflow Connect, developing and managing data pipelines with Lakeflow Spark Declarative Pipelines, and orchestrating workflows with Lakeflow Jobs. These capabilities aren’t limited to data engineers. SQL practitioners and non-technical users can tap into the same engine through interfaces like the Databricks SQL editor and Lakeflow Designer - using SQL, natural language, or visual workflows - while still benefiting from the performance, automation, and reliability of the underlying platform. With Genie Code, teams can go further with agentic data engineering, using agentic AI to generate, refine, and optimize pipelines with full awareness of their data.

Lakeflow is built for every team involved in building data pipelines, not just dedicated data engineers. SQL practitioners and other non-technical users can author, manage and troubleshoot pipelines through interfaces like the Databricks SQL editor and Lakeflow Designer, using SQL, natural language or visual workflows. At the same time, data engineering teams work on the same underlying engine, so they still get the performance, automation and reliability of the platform. This shared foundation lets organizations extend data pipeline work beyond a single specialized team.

Lakeflow combines ingestion, transformation and orchestration in a single, unified tool stack rather than requiring separate point solutions for each stage. This reduces costs and integration overhead compared with managing multiple disconnected tools, and it keeps data under built-in, unified governance and lineage throughout the pipeline. Components such as Lakeflow Connect for ingestion, Spark Declarative Pipelines for transformation, and Lakeflow Jobs for orchestration all run on the same engine, which auto-optimizes resource usage for both batch and real-time use cases.

You can get started with Lakeflow by taking the free data engineering training, a program designed to ramp up on data engineering basics in 90 minutes. From there, the Lakeflow documentation covers everything needed to start building pipelines, including ingestion, transformation and orchestration. Teams that want to see the platform in action first can also watch the data engineering demos covering the Databricks Data + AI Platform.