Sponsored by: Deloitte | Streamlined Monitoring of Databricks Workflows with Deloitte

The Databricks Data Intelligence Platform provides many tools necessary to build and run data pipelines to accommodate various data engineering tasks. With this robust suite of tools, Deloitte created a configurable and automated way to track Lakehouse data engineering tasks with minimal overhead costs in a single simplified view. Built on Databricks’ SDK and SQL dashboards, Deloitte’s Automated Workflow Monitoring solution creates a job to track all data engineering pipeline statuses in a Databricks Workspace. The solution is designed to automate large parts of the operations and maintenance of the Databricks Data Intelligence Platform and reduce the overhead costs typically associated with supplying trustworthy insights to critical data pipeline tasks. It provides a platform to build configurations that meet an enterprise’s needs and develop extensions such as customized email alerts or connections to business intelligence (BI) tools.