Agentic Data Engineering with Genie Code and Genie ZeroOps

Why data work needs more than a coding assistant

A general-purpose coding agent can write flawless Python that still produces wrong answers, because in data engineering the hard part isn't the code, it's the data. The correct definition of “revenue,” the right table among three near-duplicates, the governance policy that applies: those answers live in the catalog and its semantics, not the source code. And even a perfect pipeline is only half the job, since production assets fail loudly, drift silently, and break when upstream data changes. Purpose-built agents grounded in Unity Catalog address both halves, part of the broader shift toward agentic analytics on the Databricks Platform.

Genie Code: an AI agent that knows your data

Genie Code serves the technical users who build pipelines, notebooks, dashboards, and ML workflows. Grounded in Unity Catalog metadata, semantics, and governance policies, it can search for the right tables, propose an architecture such as a medallion pattern, generate transformation code and documentation, and validate its own work with dry runs. Adoption is already significant: 60% of Lakeflow pipelines are now authored with Genie Code, and customers report delivering three to ten times faster.

Two customization layers tailor its output to your organization:

• Instructions — an always-on markdown guide, set at the user or workspace level, that encodes universal conventions like required table properties or retention policies.
• Skills — reusable playbooks loaded only when relevant, capturing repeatable workflows such as how your company builds fraud detection pipelines or formats READMEs. Genie Code can even write new skills from a completed conversation.

An auto-approve mode checks each action against your stated intent so long-running tasks don't stall on confirmation clicks. Native connectors (such as Atlassian) and custom MCP servers bring in context beyond the catalog, so Genie Code can read a Jira ticket, build the pipeline it describes, schedule the Lakeflow job, and add the project to Git as a Declarative Automation Bundle ready for CI/CD.

Genie ZeroOps: proactive pipeline operations

Genie ZeroOps, powered by Genie Code, handles the half of data engineering that starts after deployment. It monitors jobs and schemas, separates real incidents from transient noise, and surfaces what matters in a prioritized inbox. When something breaks, it investigates without being asked: tracing a failure to the commit that caused it, validating a fix with read-only checks and dry runs in a sandbox where writes to production are blocked, and assessing downstream impact on dashboards and KPIs before ranking severity.

ZeroOps also catches silent regressions, pairing ML-based anomaly detection with agentic validation to confirm whether a sudden metric drop is a genuine defect or a false positive. Every remediation lands as a pull request, often with new tests to prevent the same regression, and a human review remains the final gate. Over time it learns your preferences, remembering which assets matter and which noise to ignore.

Faster authoring demands automated operations

Accelerated authoring multiplies pipelines, and every new pipeline adds operational load. Pairing Genie Code at build time with Genie ZeroOps in production is the scaling answer: teams ship more assets with confidence while agents absorb the toil of keeping them correct and compliant. It's a practical picture of how AI is transforming data analytics — agents moving from passive assistants to accountable, human-in-the-loop teammates.

Availability

Genie Code's pipeline construction and scheduling capabilities are available today, connectors are in beta, and the full-page Genie Code experience is in beta. Genie ZeroOps is expected to enter private preview soon.