Session

Migrate to Lakeflow Jobs: Modernizing Data Orchestration on Databricks

Overview

ExperienceIn Person
TrackData Engineering & Streaming
IndustryManufacturing, Travel & Hospitality, Financial Services
TechnologiesLakeflow
Skill LevelBeginner

Still running orchestration outside Databricks? That often means more tools to manage, less context about your data, and more overhead connecting pipelines, schedules, and dependencies across systems.

 

Join this session to learn how to migrate to Lakeflow Jobs, Databricks’ native orchestrator for data and AI workloads. We will cover practical migration patterns, the tooling available to accelerate the move, and the tips, techniques, and best practices that help teams reduce risk and avoid common pitfalls.

 

You will learn:

  • Why teams are consolidating orchestration onto Databricks
  • How common migration patterns from legacy orchestrators map to Lakeflow Jobs
  • What tooling, tips, and techniques can speed up migration and simplify rollout
  • The best practices that help teams move faster with less operational overhead
  • What real customer migrations look like in practice, including key challenges, tradeoffs, and outcomes

Session Speakers

Greg Hansen

/Specialist Solutions Architect
Databricks

Saad Ansari

/Product Manager
Databricks

Full Summary

Migrating legacy orchestration to Lakeflow Jobs without the pain

Organizations rarely swap orchestrators until scale, cost, reliability, or platform shifts force the issue. The session unpacked why those moves happen and how agent-assisted tooling changes the migration math to Lakeflow Jobs, the orchestrator built into the Databricks Platform. Rather than a feature pitch, the focus stayed on intent-preserving migration that reduces switch costs while modernizing patterns.

FAQ


No. The tool does not extract credentials from external secret stores. It generates a setup script to create a secret-backed scope in Databricks, giving you a clean location to store credentials and reference them in migrated jobs.

The converter generates a placeholder notebook for the unsupported activity and flags it for follow-up before deployment. The team is also building a framework to add custom integrations, with Genie skills to help author them.

Bundles are version-controlled configuration. Teams can review changes in Git, promote across dev, staging, and prod, and edit outputs as needed. They also keep results inspectable rather than a black-box deployment.

Not by default. Skills must currently be installed. The plan is to package and deliver them via the Lakebridge product so Genie can invoke end-to-end skills for orchestration, code, and SQL migrations.

Expect light touch-ups. Common tasks include setting target catalogs and schemas, running generated setup notebooks to create prerequisites like Unity Catalog connections, and reviewing the profiling report. The translation of activities, parameter mapping, and pattern modernization are handled by the agent and deterministic converters.