Session

Sponsored by: Accenture | From Pipelines to Agents: Rethinking Data Engineering for Autonomous Systems

Overview

ExperienceIn Person
TrackData Engineering & Streaming
IndustryHealthcare & Life Sciences, Manufacturing, Financial Services
TechnologiesDelta Sharing, Lakeflow, Lakebase
Skill LevelIntermediate

The next evolution of data engineering isn’t incremental—it’s autonomous. In this session, we challenge the traditional notion of pipelines and introduce an agentic model where data systems can reason, adapt, and act. Using Databricks as the foundation, we explore how agentic AI enables dynamic orchestration, intelligent error handling, and continuous optimization across the data lifecycle to build and run a complete data intelligence platform. We’ll share practical design patterns, lessons learned, and the trade-offs organizations must navigate as they move toward self-evolving data platforms—balancing autonomy with control, and innovation with governance.

Session Speakers

Abderrahmane FILALI

/Data architecture Lead
Avanade

Chintan Shah

/Senior Manager
Avanade

Full Summary

Reimagining data engineering for the agentic AI era

Data engineering is straining under rising data volume and complexity, prolonged manual workflows, and slow time to insight. The conversation reframes the discipline for an agentic AI world, arguing that specialized agents can take over deterministic, repeatable tasks while humans stay in control of strategy and governance on the Databricks Platform. Teams that adopt this approach see meaningful efficiency gains, with about 40 percent of a typical engineering week reclaimed for higher value work, a shift explored further in how AI is transforming data engineering pipelines.

FAQ


No. Agents handle deterministic, repeatable tasks such as ingestion setup, standardization, and initial transformation logic. Data engineers review pull requests, approve configurations, maintain semantics, and guide platform strategy. Human oversight is required before anything is promoted.

Agents commit generated YAML or code to a repository and always open pull requests. Engineers review and merge changes, and Databricks asset bundles control environment promotion. MLflow logging provides traceability for every agent session and external tool call.

Agents can build syntactically correct pipelines without understanding business meaning. The ontology provides concepts, relationships, and links from conceptual models to physical tables, enabling agents to produce domain-aligned data products, such as a customer 360, that match how the business defines metrics.

Unity Catalog provides governance and lineage. Lakeflow Connect and Auto Loader power ingestion. Lakehouse monitoring supplies data quality signals. MLflow captures observability, and Databricks asset bundles manage deployment. Genie supports conversational assistance across the platform.

Gains come from automating repeatable, deterministic tasks and agent-assisting partially automatable work. A typical week includes about 13 hours that agents can fully automate plus roughly 3 hours aided by agents, totaling about 40 percent time saved for engineers.