Human-in-the-Loop Compliance at Global Scale: Databricks Apps + Lakebase Casework + SME Labeling
Overview
| Experience | In Person |
|---|---|
| Track | Application Development |
| Industry | Enterprise Technology, Consulting & Services, Financial Services |
| Technologies | Databricks Apps, Databricks Agents, Lakebase |
| Skill Level | Intermediate |
| DOWNLOAD SESSION SLIDES | |
In this session, we share a practical pattern we call “PolicyOps” on Databricks: converting long policy documents into cited, versioned, queryable policy units, then operationalizing a human‑in‑the‑loop workflow that captures domain SME judgment as structured labels. We’ll show how to parse and structure large policy PDFs into clause-level records (including page-aware citations), how to extract decision criteria into queryable tables, and how to connect policy interpretation to real operational review.We’ll then demonstrate an SME review queue built with Databricks Apps — designed for high-signal feedback (not just thumbs up/down) — with Lakebase used to manage case workflow state (assignment, status, reviewer actions, approvals). The result is a scalable workflow that keeps humans in control of edge cases while systematically converting implicit SME knowledge into reusable labels and corrections that improve quality over time.
Session Speakers
Salman Chaudhry
/VP AI Engineering
Mastercard
Full Summary
From POC to production: how Mastercard built auditor-ready agentic AI with policy ops
Enterprises in regulated industries struggle to turn agentic AI proofs of concept into systems that generate ROI and withstand audits. The talk outlines a practical operating model that makes decisions explainable, reproducible, and safe by elevating "policy ops" as the discipline for shipping defensible outcomes, not just models, using the Databricks Platform. Teams looking to understand the broader shift this represents can explore how AI is transforming data analytics across regulated and high-stakes domains.
FAQ
MLOps focuses on shipping and monitoring models. Policy ops focuses on shipping defensible decisions in regulated processes. The primitives are cases, traces, and structured SME labels, with the goal of producing outcomes an auditor can follow end to end.
Non-technical reviewers found engineering tools unfamiliar and overwhelming. A purpose-built app, designed with UX research, presents prioritized cases, summarized traces, and cited clauses at the level where SMEs can act.
Policies are parsed into clause-level records using a defined schema, with unique IDs, page references, and version lineage. Chunking is done for accountability rather than embedding length, preserving traceability for auditing and review.
Reviewer actions are written as structured labels, synced to Delta Lake, and incorporated into evaluation datasets. Depending on the error type, teams tune models or expand evaluation coverage so similar cases are caught before production.
No. Agents assemble evidence and draft rationales, while SMEs make final decisions and handle exceptions. The approach aims to scale expert impact, not reduce expert roles.