Session

Human-in-the-Loop Compliance at Global Scale: Databricks Apps + Lakebase Casework + SME Labeling

Overview

ExperienceIn Person
TrackApplication Development
IndustryEnterprise Technology, Consulting & Services, Financial Services
TechnologiesDatabricks Apps, Databricks Agents, Lakebase
Skill LevelIntermediate
DOWNLOAD SESSION SLIDES

In this session, we share a practical pattern we call “PolicyOps” on Databricks: converting long policy documents into cited, versioned, queryable policy units, then operationalizing a human‑in‑the‑loop workflow that captures domain SME judgment as structured labels. We’ll show how to parse and structure large policy PDFs into clause-level records (including page-aware citations), how to extract decision criteria into queryable tables, and how to connect policy interpretation to real operational review.We’ll then demonstrate an SME review queue built with Databricks Apps — designed for high-signal feedback (not just thumbs up/down) — with Lakebase used to manage case workflow state (assignment, status, reviewer actions, approvals). The result is a scalable workflow that keeps humans in control of edge cases while systematically converting implicit SME knowledge into reusable labels and corrections that improve quality over time.

Session Speakers

Salman Chaudhry

/VP AI Engineering
Mastercard

Full Summary

From POC to production: how Mastercard built auditor-ready agentic AI with policy ops

Enterprises in regulated industries struggle to turn agentic AI proofs of concept into systems that generate ROI and withstand audits. The talk outlines a practical operating model that makes decisions explainable, reproducible, and safe by elevating "policy ops" as the discipline for shipping defensible outcomes, not just models, using the Databricks Platform. Teams looking to understand the broader shift this represents can explore how AI is transforming data analytics across regulated and high-stakes domains.

FAQ


MLOps focuses on shipping and monitoring models. Policy ops focuses on shipping defensible decisions in regulated processes. The primitives are cases, traces, and structured SME labels, with the goal of producing outcomes an auditor can follow end to end.

Non-technical reviewers found engineering tools unfamiliar and overwhelming. A purpose-built app, designed with UX research, presents prioritized cases, summarized traces, and cited clauses at the level where SMEs can act.

Policies are parsed into clause-level records using a defined schema, with unique IDs, page references, and version lineage. Chunking is done for accountability rather than embedding length, preserving traceability for auditing and review.

Reviewer actions are written as structured labels, synced to Delta Lake, and incorporated into evaluation datasets. Depending on the error type, teams tune models or expand evaluation coverage so similar cases are caught before production.

No. Agents assemble evidence and draft rationales, while SMEs make final decisions and handle exceptions. The approach aims to scale expert impact, not reduce expert roles.