Stop Guessing, Start Scoring: LLM Extraction Evaluation on Databricks
Overview
| Experience | In Person |
|---|---|
| Track | Artificial Intelligence & Agents |
| Industry | Consulting & Services |
| Technologies | Databricks Apps, Databricks Agents, Lakebase |
| Skill Level | Intermediate |
| DOWNLOAD SESSION SLIDES | |
Session Speakers
Darshana Nair
/Data Scientist Director
CLA
Michelle JanneyCoyle
/AI Forward Deployed Engineer
Databricks
Full Summary
From guesswork to measurable trust: how CLI built an evaluated SOC extraction agent on Databricks
Audit teams spend millions of hours pulling structured facts from long, inconsistent SOC (System and Organization Control) reports. Large language models can accelerate extraction, but without rigorous evaluation, quality becomes a matter of faith. The session outlines how an end-to-end workflow on the Databricks Platform pairs Databricks document AI capabilities with measurement and human review to move from hopeful prototypes to trustworthy production.
FAQ
A SOC report documents the controls a service organization has in place. Reports often span 50 to 100 pages with highly variable, unstructured sections such as control objectives and user entity controls. Inconsistent formatting across firms makes automated, consistent extraction difficult.
Confidence scores come from an LLM judge at runtime and do not require ground truth. They serve as triage, guiding reviewer attention and providing operational signals. Accuracy metrics such as precision, recall, and F1 are computed later by comparing model output to auditor-corrected values to track true performance over time.
No. The process is explicitly human in the loop. The agent extracts and scores fields, auditors review and correct, and their decisions become structured feedback that drives continuous improvement without removing professional judgment.
An aggregate score conceals where the model is strong or weak. Scoring by section, object, and field identifies specific failures, supports targeted fixes, and provides reliable run-over-run comparisons to confirm whether changes improved quality.
The SOC workflow is linked to a managed MLflow experiment that captures traces and evaluation metrics. Those logs underpin the operations dashboard for latency, token usage, cost, and quality, and enable comparisons across runs to validate that prompt, model, or workflow changes deliver measurable gains.