Session

Agentic Data Engineering: Automating Pipelines with Genie Code

Overview

Experience	Virtual
Technologies	Lakeflow, Genie

Genie Code adds tremendous value for data engineers - unifying and automating much of the ingest, ETL and streaming process. Hear from Bilal Aslam (Senior Director of Product Management) and our customer SEGA on how to use Genie Code for agentic data engineering, automating much of the process of building intelligent data pipelines. It integrates into the broader Databricks environment, and with Lakeflow and Spark Declarative Pipelines is a fully governed way to prepare and analyze data directly in Databricks.

Session Speakers

Amber Roberts

/Staff Tech Marketing Engineer
Databricks

Anastasiya Lavrenchuk

/Data Developer
SEGA Europe

Bilal Aslam

/Sr. Director of Product Management
Databricks

Maria Zervou

/Chief AI Officer - EMEA
Databricks

Full Summary

Inside Genie Code and Genie ZeroOps: How agentic AI is reshaping data engineering

A focused discussion explores how agentic AI is changing the way data teams build and operate pipelines on the Databricks Platform. The conversation centers on two agents, Genie Code for creation and Genie ZeroOps for operations, and why declarative design is the right substrate for safe automation.

Teams looking for grounding on what this shift means in practice will find relevant context in the Databricks data engineering documentation.

The core message is pragmatic: agents that understand data engineering context move work from autocomplete and snippets to end‑to‑end workflows built and run to standard, freeing humans to handle decisions and domain judgment.

An AI coding agent built for data engineering

Genie Code is positioned as a specialized coding agent for Databricks, aware of workspace context and the realities of data engineering and machine learning. It aims to generate production‑grade artifacts, not just lines of code. A single prompt can scaffold Databricks Notebooks, Lakeflow Declarative Pipelines, and Lakeflow Jobs that reflect established patterns like the medallion architecture, incorporate data quality expectations, and include tests.

That framing lowers the barrier to productive starts. Newer practitioners can begin with a solid baseline instead of mastering every platform best practice first, then iterate with edits or follow‑up prompts. The agent works at any granularity, from creating a pipeline from scratch to modifying a single step or adding validation to existing code.

From faster builds to autonomous operations

Accelerating pipeline creation increases the surface area to operate. Genie ZeroOps addresses that pressure by running in the background, watching pipelines and data, assembling context, and surfacing issues before or as they occur. Routine toil, like collecting logs and diagnosing common failures, is a target for automation so that when human intervention is necessary, the triage is already done.

The ambition extends beyond data pipelines. The speakers note that additional areas, including machine learning, are in scope over time, guided by a simple philosophy: wherever toil dominates today, automation should follow.

Declarative pipelines as an agent multiplier

Lakeflow Declarative Pipelines are highlighted as a natural fit for agentic workflows. Because the framework captures the desired state rather than imperative steps, agents can focus on intent, while the system determines execution details. That division of labor improves reliability and makes changes safer in production.

A key operational benefit is clean rollback. When the target state is declared once, the system can return to it predictably, which is harder to guarantee with imperative code paths. Agents can then safely propose and apply updates while the platform ensures consistent outcomes.

A practitioner snapshot: Sega Europe's migration

A real‑world account from Sega Europe underscores the shift. After moving from a legacy Spark streaming setup to Lakeflow Declarative Pipelines in late 2024, launch‑day support dropped from multiple developers working around the clock to a single developer spending a few hours with studios. The team stress‑tested the new approach during the Football Manager 2026 release and reported a smooth result.

Governance remains central. User groups and permissions segment access so each audience sees only the data intended for them. For teams cautious about agent‑generated changes, a practical pattern emerges: use planning mode to preview an agent's intended steps, keep peer review in the loop, and treat the agent as a force multiplier rather than a black box.

Why the combination matters now

Agentic coding paired with declarative pipelines compresses the path from idea to trusted production. Teams ship faster, operate more predictably, and reserve human time for judgment calls. The result is greater throughput with less toil, higher confidence, and smaller on‑call rotations.

FAQ

From the Databricks home page, click the code button to open Genie Code, describe what you want, and the agent begins generating Databricks Notebooks, Lakeflow Declarative Pipelines, and Lakeflow Jobs. You can then edit the artifacts directly or iterate with follow‑up prompts.

Genie Code focuses on building pipelines faster and to standard. Genie ZeroOps is autonomous and runs in the background to monitor pipelines and data, assemble context, and catch issues before or as they happen. Think build versus continuous operations.

It can do both. The agent can scaffold a pipeline from scratch, update a specific step, add tests, or extend an existing workflow, allowing incremental adoption without wholesale rewrites.

Declarative frameworks capture the what, allowing agents to express intent and let the system handle the how. That reduces complexity and improves safety, including straightforward rollbacks that are harder with imperative systems.

Use planning mode so the agent reveals its intended actions before execution, keep human review in the loop, and treat generated outputs as a starting point. That approach balances speed with the quality controls production systems require.