Session

The Right Model for the Job: A Practical Framework for AI

Overview

Experience	In Person
Track	Artificial Intelligence & Agents
Industry	Enterprise Technology
Technologies	Databricks Agents
Skill Level	Intermediate
DOWNLOAD SESSION SLIDES

Not every problem needs the biggest model — and not every model works out of the box. With frontier models, small open models, and fine-tuning all on the table, the hardest question isn't "can we do AI?" It's "which model is actually right for this?"In this session, we walk through how to match models to use cases across real enterprise workloads. You'll see when a frontier model earns its cost, when a small open model wins on latency, and when fine-tuning turns a generic model into a specialist that knows your business. We'll put the tradeoffs in action — quality, cost, latency, control — using concrete examples across customer support, document processing, and analytics.You'll leave with a repeatable framework for choosing, customizing, and deploying the right model on Databricks every time — and the patterns to keep those decisions sharp as new models land.

Session Speakers

Nuno Castro

/Director AI FDE
Databricks

Viktoria Semaan

/Principal Technical Evangelist
Databricks

Full Summary

Right model, right job: a practical framework for choosing and swapping LLMs

Enterprises are moving from proof of concept to production with generative AI, and the economics are changing. Wiring everything to a single frontier API no longer scales as usage grows and token bills spike. The session lays out a pragmatic, three-step framework to pick the right model per task, govern models in production, and fine-tune only when the data and volume justify it, using MLflow, Unity AI Gateway, and Databricks AI Runtime on the Databricks Platform.

Model choice becomes an engineering discipline, not a brand decision. Benchmarks change monthly, open-weight models are catching up fast, and many organizations already run multiple models side by side. A repeatable process, backed by evaluation and governance, lets teams swap models without rewrites and pay only where quality returns value. Teams navigating this shift can explore how enterprise AI agent trends are reshaping governance, evaluations, and use cases as a useful frame for what production-grade model operations now require.

Why the model landscape demands a new operating model

Top-ranked LLMs now turn over in roughly two months, and capabilities are doubling on a seven-month cadence. Open-weight models trail proprietary leaders by about three and a half months on intelligence, and more than a third of enterprises already operate five or more models in production. A similar share of token volume now comes from open weights.

Real-world economics illustrate the shift. A subscription-based coding assistant, Courser, started with proprietary APIs and saw unit economics break as usage scaled. By fine-tuning an open-weight Kimi model on its coding tasks, the team held quality while cutting tokens per task by 86 percent. Platform lock-in is turning into a business risk rather than a simplification.

Build the evaluation before you pick the model

Start by defining what good looks like. With MLflow, teams assemble an evaluation set, edge cases, and scorers, which can be simple Python functions, heuristics, guidelines, or LLM judges. One run produces traces per row and comparable metrics across candidates, so swapping models or prompts becomes a one-line change and an apples-to-apples comparison.

Evaluation runs in two reinforcing modes. Automated runs provide speed and scale whenever a model or prompt changes. Human review, through a dedicated app, catches nuances automation misses and writes corrections back into the dataset, raising the ground truth over time. In a sports chat moderation demo, Opus 4.8 led a classification task at 0.94 accuracy, but at roughly 3,600 dollars per million messages. A nano-class model cost about 9 dollars, creating a 40x price gap and a clear tradeoff to weigh.

Govern multi-model production and keep swapping safely

Unity AI Gateway serves as a single control plane for all models, coding tools, and MCP servers an organization touches. Teams add endpoints for new models, split traffic across candidates, configure fallbacks, set rate limits by group or user, and observe live traces diverging from training. A prebuilt dashboard surfaces usage, cost, and performance, and a natural language interface helps answer questions such as who is overspending on premium models.

Decouple application code from the serving layer by routing calls through the gateway. Start with the smallest model that clears the quality bar, then climb only when evaluation requires it. As new models drop, send a small share of traffic to them and compare results under the same yardstick.

Fine-tune only when the task earns it

Fine-tuning is the last resort, valuable when three signals align:

Errors are repetitive and patterned.
You have thousands of labeled input‑output pairs.
Prompting and retrieval have been maxed out.

Databricks AI Runtime brings serverless GPUs that scale to zero, so parameter-efficient fine-tunes on small open models become quick notebook jobs. Tracked in MLflow, registered in Unity Catalog, and deployed to serving, these specialist models can deliver big wins where volume is high and patterns are stable. In the demo, a fine-tuned Qwen 1.7B reached 91 percent classification accuracy versus Opus 4.8 at 94 percent, at about 8 dollars per million messages instead of 3,600, and with lower latency. The same approach underperformed on a reasoning task, scoring 1.3 out of 5, while Sonnet reached 4.7 and Opus 4.5. Paying for a frontier model made sense there because it was invoked only about 1 percent of the time.

Operationalize model choice as a continuous loop

Decompose workloads into jobs, define a quality bar you own, evaluate systematically, govern the mix through a gateway, and fine-tune only where the data and volume earn the complexity. No model wins every axis or forever, so let a repeatable process, not brand loyalty, drive decisions.

FAQ

Fine-tuning is warranted when mistakes are repetitive, you have a few thousand labeled examples, and you have exhausted prompting or retrieval. It reduces context size and cost per call but adds maintenance overhead, so reach for it only after evaluation shows clear upside.

Route calls through Unity AI Gateway. Add new models as endpoints, split a small percentage of traffic to candidates, set fallbacks and rate limits, and compare results under the same evaluation suite. Application code stays decoupled from model changes.

Automated runs trigger programmatically whenever a model or prompt changes, enabling speed and scale. Human review inspects traces for nuanced errors and writes corrections back into the dataset, improving the ground truth and the evaluation over time.

Not necessarily. In the demo, Opus 4.8 led on accuracy for classification but at about 3,600 dollars per million messages, while a nano-class option cost roughly 9 dollars. The right choice depends on whether the marginal accuracy moves your business outcome.

AI Runtime provides serverless GPU compute that scales to zero, so you pay only for the seconds a job runs. You attach a GPU to a notebook, run LoRA or QLoRA, track in MLflow, register in Unity Catalog, and deploy to serving without managing infrastructure.