Databricks Workflows
Unified orchestration for data, analytics and AI on the Data Intelligence Platform
Databricks Workflows is a managed orchestration service, fully integrated with the Databricks Data Intelligence Platform. Workflows lets you easily define, manage and monitor multitask workflows for ETL, analytics and machine learning pipelines. With a wide range of supported task types, deep observability capabilities and high reliability, your data teams are empowered to better automate and orchestrate any pipeline and become more productive.
Simple Authoring
Whether you’re a data engineer, a data analyst or a data scientist, easily define workflows with just a few clicks or use your favorite IDE.
Actionable Insights
Get full visibility into each task running in every workflow and get notified immediately on issues that require troubleshooting.
Proven Reliability
Having a fully managed orchestration service means having the peace of mind that your production workflows are up and running. With 99.95% uptime, Databricks Workflows is trusted by thousands of organizations.
How does it work?
Unified with the Databricks Data Intelligence Platform
Reliability in production
Deep monitoring and observability
Batch and streaming
Efficient compute
Seamless user experience
Unified with the Databricks Data Intelligence Platform
Unlike external orchestration tools, Databricks Workflows is fully integrated with the Databricks Data Intelligence Platform. This means you get native workflow authoring in your workspace and the ability to automate any platform capability including Delta Live Tables pipelines, Databricks notebooks and Databricks SQL queries. With Unity Catalog, you get automated data lineage for every workflow so you stay in control of all your data assets across the organization.
Reliability at scale
Every day, thousands of organizations trust Databricks Workflows to run millions of production workloads across AWS, Azure and GCP with 99.95% uptime. Having a fully managed orchestration tool built into the Data Intelligence Platform means you don’t need to maintain, update or troubleshoot another separate tool for orchestration.
Deep monitoring and observability
Full integration with the Data Intelligence Platform means Databricks Workflows provides you with better observability than any external orchestration tool. Stay in control by getting a full view of every workflow run and set notifications for failures to alert your team via email, Slack, PagerDuty or a custom webhook so you can get ahead of issues quickly and troubleshoot before data consumers are impacted.
Batch and streaming
Databricks Workflows provides you with a single solution to orchestrate tasks in any scenario on the Data Intelligence Platform. Use a scheduled workflow run for recurring jobs that do batch ingestion on preset times or implement real-time data pipelines that run continuously. You can also set a workflow to run when new data is made available using file arrival triggers.
Efficient compute
Orchestrating with Databricks Workflows gives you better price/performance for your automated, production workloads. Get significant cost savings when utilizing automated job clusters that have a lower cost and are only running when a job is scheduled so you don’t pay for unused resources. In addition, shared job clusters let you reuse compute resources for multiple tasks so you can optimize resource utilization.
Seamless user experience
Define workflows in your preferred environment — easily create workflows right in the Databricks workspace UI or using your favorite IDE. Define tasks that use a version-controlled notebook in a Databricks Repo or in a remote Git repository and adhere to DevOps best practices such as CI/CD.
