Databricks Apps: A Magic Wand for Driving Data Quality Adoption and Observability
Overview
| Experience | In Person |
|---|---|
| Track | Application Development |
| Industry | Communications - Media & Entertainment |
| Technologies | Databricks SQL, Unity Catalog, Databricks Apps |
| Skill Level | Intermediate |
Most data quality tools don’t fail because the checks are wrong. They fail because users don’t understand how to use them.In this talk, we’ll share how we rebuilt our internal data quality platform using Databricks Apps and Databricks Workflows table triggers to prioritize usability and holistic visibility. We moved away from a config-file-driven model owned by a small group of platform engineers toward a user-centric approach that empowers teams to create, test and manage their own data quality rules.
With a Databricks App as the central interface, users can define and test rules, inspect production run history, and track data quality trends across the tables and teams they own. Combined with table-triggered workflows, checks run automatically as data changes without added operational complexity.This session covers our architecture, key design decisions, and lessons learned, and shows how Databricks Apps can turn data infrastructure into a user-friendly platform.
Session Speakers
Trinity Xia
/Sr Software Engineer
Scribd, Inc.