Session

Stop Guessing, Start Scoring: LLM Extraction Evaluation on Databricks

Overview

ExperienceIn Person
TrackArtificial Intelligence & Agents
IndustryConsulting & Services
TechnologiesDatabricks Apps, Databricks Agents, Lakebase
Skill LevelIntermediate
DOWNLOAD SESSION SLIDES
Speakers: Michelle JanneyCoyle (Databricks), Darshana Nair (CLA)Robust evaluation is key to building reliable LLM-based systems. Together with CliftonLarsonAllen (CLA), a leading accounting and professional services firm, we developed a Databricks-native evaluation solution. This system centralizes metrics, ground truth, and expert feedback to enable repeatable offline evaluation for their SOC extraction workflow.In this talk, we deconstruct our implementation, highlighting how we use managed MLflow, custom scorers, and a Databricks App with Lakebase to capture SME feedback and structured ground truth. During this talk you will learn how this framework makes quality measurable and provides clear signal for future improvements. Key TakeawaysAutomating evaluation using Databricks tools.Using Databricks Apps to bridge the gap between Data Science and domain experts.Lessons from evaluating complex semantic extraction and structured JSON outputs.

Session Speakers

Darshana Nair

/Data Scientist Director
CLA

Michelle JanneyCoyle

/AI Forward Deployed Engineer
Databricks

Full Summary

From guesswork to measurable trust: how CLI built an evaluated SOC extraction agent on Databricks

Audit teams spend millions of hours pulling structured facts from long, inconsistent SOC (System and Organization Control) reports. Large language models can accelerate extraction, but without rigorous evaluation, quality becomes a matter of faith. The session outlines how an end-to-end workflow on the Databricks Platform pairs Databricks document AI capabilities with measurement and human review to move from hopeful prototypes to trustworthy production.

FAQ


A SOC report documents the controls a service organization has in place. Reports often span 50 to 100 pages with highly variable, unstructured sections such as control objectives and user entity controls. Inconsistent formatting across firms makes automated, consistent extraction difficult.

Confidence scores come from an LLM judge at runtime and do not require ground truth. They serve as triage, guiding reviewer attention and providing operational signals. Accuracy metrics such as precision, recall, and F1 are computed later by comparing model output to auditor-corrected values to track true performance over time.

No. The process is explicitly human in the loop. The agent extracts and scores fields, auditors review and correct, and their decisions become structured feedback that drives continuous improvement without removing professional judgment.

An aggregate score conceals where the model is strong or weak. Scoring by section, object, and field identifies specific failures, supports targeted fixes, and provides reliable run-over-run comparisons to confirm whether changes improved quality.

The SOC workflow is linked to a managed MLflow experiment that captures traces and evaluation metrics. Those logs underpin the operations dashboard for latency, token usage, cost, and quality, and enable comparisons across runs to validate that prompt, model, or workflow changes deliver measurable gains.