Behind the Curtain: How We Do Eval in Genie

The talk outlines how Genie is engineered to prioritize grounded correctness over speed, with evaluation treated as core product work central to AI/BI Dashboards and intelligent analytics on real-world data.

Genie stays inside a customer's domain, prefers "I do not know" to a guess, and inherits Unity Catalog permissions so queries never exceed a user's access. The guiding idea is simple: enterprises need answers that are right, not answers that sound right.

Redefining accuracy for enterprise questions

Accuracy begins with faithful schema understanding. Misread metadata does not trigger errors, it quietly produces valid SQL and convincing charts with broken logic. Genie is built to avoid that failure mode by grounding every step in the customer's business context, asking for clarification when prompts are vague, and refusing to guess when evidence is thin.

Governance is integral to correctness. Unity Catalog permissions constrain what Genie can see and query, so generated answers respect the same access controls as the underlying data.

Building ground truth and evaluations that mirror reality

Evaluation must reflect real usage, not synthetic prompts. Production traffic analysis informs benchmarks across four workflow patterns, from standard metric reporting to multi‑turn strategic analysis, and across four difficulty tiers, from simple aggregations to open‑ended expert questions. That distribution keeps offline tests representative of enterprise behavior.

Ground truth comes from three sources. Academic benchmarks such as Spider and BIRD are mapped to functional categories to validate structural logic. Internal experts craft cases around messy schemas, non‑standard naming, and business‑specific rules. External vendors provide blind audits and large‑scale labeled query pairs to reduce internal bias. LLM‑as‑judge extends coverage for complex tasks where exact match metrics fail, with the caveat that judges show a 10 to 20 percent positive bias on numeric tasks. Their value lies in consistent relative deltas across model versions, which makes pairwise comparisons reliable during optimization.

Guardrails, audits, and regression protection in production

Continuous rigor underpins every release. Large real‑world benchmarks run for each model or performance update to catch regressions early. Multi‑layered auditing pairs LLM grading with human experts to keep the benchmark ground truth trustworthy. Adaptive guardrails govern live interactions, combining permission checks with clarification prompts when a question is too ambiguous to answer deterministically.

A standing regression guard includes more than 2,000 golden queries, thousands of adversarial ambiguity tests, and multi‑turn conversation flows. The evaluation platform executes the exact production code path in a sandbox to eliminate testing gaps, runs daily high‑coverage benchmarks, and triggers real‑time alerts when scores fall below thresholds so engineering can intervene before users are impacted.

Handling non‑determinism and learning from real usage

Business reporting demands repeatability, yet LLMs are probabilistic. Genie reduces variance with concrete few‑shot examples, narrowly scoped instructions and SQL patterns, and clarification prompts when a question cannot be answered deterministically. That design favors stable, defensible outputs over fast but fragile guesses.

Offline tests can grow stale, so production signals drive continuous improvement. Customer‑flagged failures are root‑caused and promoted to permanent tests. Exporting data and sharing reports serve as strong positive signals that answers were trusted, while thumbs‑up and thumbs‑down feedback highlights friction points. Sampling noise and novelty effects can swing single measurements by about 10 percent, so progress is validated with strict confidence intervals rather than single runs.

Evaluation is the backbone of trustworthy conversational BI

Enterprises cannot outsource judgment to plausibility. By engineering representative ground truth, pairing human and LLM grading, enforcing governance and ambiguity guardrails, and validating gains with statistical rigor, Genie turns evaluation into a product capability. Teams get speed without guesswork, and numbers they can defend.