Session

Scaling Custom LLMs with vLLM and Databricks Model Serving: Fast, Flexible, and Production-Ready

Overview

ExperienceIn Person
TrackArtificial Intelligence & Agents
IndustryConsulting & Services
TechnologiesDatabricks Agents
Skill LevelAdvanced
DOWNLOAD SESSION SLIDES

Databricks Model Serving supports deployments ranging from classical ML models on custom-CPU workloads to foundation models with dedicated provisioned throughput endpoints. But what about the use-cases that need any of the 1000s of other open-source or fine-tuned LLMs? Deploying them efficiently can be challenging.

This breakout session explores deploying LLMs on custom-GPU endpoints with vLLM. In it, we’ll examine:

  • Serverless GPU Compute: How it simplifies the deployment process, saving hours on configuration work alone.
  • vLLM + GPU workloads: How they work in tandem on the platform to deliver high-throughput inference with scalable infrastructure.
  • Implementation: A step-by-step code walkthrough for packaging models, configuring vLLM runtime, and deployment!

Serving LLMs on GPUs doesn’t have to be scary. Learn more about how Databricks enables you to deploy even your most demanding model serving workloads!

Session Speakers

Speaker placeholderIMAGE COMING SOON

Colton Peltier

/Senior Staff AI FDE
Databricks

Mohamad Aboufoul

/Senior AI FDE
Databricks

Full Summary

Serving custom LLMs on Databricks with vLLM: A new path for specialized models

Running large language models in production often turns into an infrastructure project, not a product one. GPU scheduling, batching, caching, and runtime tuning all add complexity. A new beta capability brings vLLM-backed custom model hosting to Databricks Model Serving, creating a supported route to deploy small, open source, or self-trained LLMs on managed GPUs while retaining governance and scale. Teams that have been limited by provisioned throughput families now have a way to serve models that fall outside the catalog, without leaving the Databricks Platform.

FAQ


Chat completion models are supported, including multimodal vision-language models. The demo flow applied to a Qwen vision-language checkpoint. Embedding models are not currently supported.

Provisioned throughput covers specific supported families and their fine-tuned variants. The vLLM-based path is designed for everything outside that list, such as small open source models or custom-trained checkpoints, while keeping the same managed serving experience.

Endpoint provisioning can take 20 to 30 minutes. A quick local run in the notebook confirms the model loads, responds, and respects your configuration before you wait for a full endpoint build.

Avoid temporary file paths for weights, verify permissions so others can access the model, and increase the Databricks SDK deployment timeout beyond the 20-minute default for larger GPU endpoints.

Access is enabled at the workspace level. Ask your workspace administrator to turn it on, then follow the documentation and sample notebooks referenced in the session to configure your first custom vLLM endpoint.