Built on an open architecture, AI and machine learning on Databricks empowers ML teams to prepare and process data, streamlines cross-team collaboration and standardizes the full ML lifecycle from experimentation to production including for generative AI and large language models.
.png)
Simplify all aspects of data for AI and ML
Because Databricks ML is built on an open foundation with Delta Lake, you can empower your machine learning teams to access, explore and prepare any type of data at any scale. Turn features into production pipelines in a self-service manner using a few clicks without depending on data engineering support.
Automate experiment tracking and governance
Managed MLflow automatically tracks your experiments and logs parameters, metrics, versioning of data and code, as well as model artifacts with each training run. You can quickly see previous runs, compare results and reproduce a past result, as needed. Once you have identified the best version of a model for production, register it to the Model Registry to simplify handoffs along the deployment lifecycle.
Manage the full model lifecycle from data to production — and back
Once trained models, including large language models, are registered, you can collaboratively manage them through their lifecycle with the Model Registry. Models can be versioned and moved through various stages, such as experimentation, staging, production and archived. The lifecycle management integrates with approval and governance workflows according to role-based access controls. Comments and email notifications provide a rich, collaborative environment for data teams.
Deploy ML models at scale and low latency
Deploy models with a single click without having to worry about server management or scale constraints. With Databricks, you can deploy your models as REST API endpoints anywhere with enterprise-grade availability.
Use generative AI and large language models
Databricks allows you to start with an existing large language model (LLM) like Llama 2, MPT, BGE, Azure OpenAI, or Anthropic, and augment or fine-tune it with your enterprise data or build your own custom LLM from scratch through pretraining. Any existing LLMs can be deployed, governed, queried and monitored. We make it easy to extend these models using techniques like retrieval augmented generation (RAG), parameter-efficient fine-tuning (PEFT) or standard fine-tuning.
Product components
Collaborative Notebooks
Databricks Notebooks natively support Python, R, SQL and Scala so practitioners can work together with the languages and libraries of their choice to discover, visualize and share insights.
Runtime for Machine Learning
One-click access to preconfigured ML-optimized clusters, powered by a scalable and reliable distribution of the most popular ML frameworks (such as PyTorch, TensorFlow and scikit-learn), with built-in optimizations for unmatched performance at scale.
Feature Store
Facilitate the reuse of features with a data lineage–based feature search that leverages automatically logged data sources. Make features available for training and serving with simplified model deployment that doesn’t require changes to the client application.
AutoML
Empower everyone from ML experts to citizen data scientists with a “glass box” approach to AutoML that delivers not only the highest performing model, but also generates code for further refinement by experts.
Managed MLflow
Built on top of MLflow — the world’s leading open source platform for the ML lifecycle — Managed MLflow helps ML models quickly move from experimentation to production, with enterprise security, reliability and scale.
Production-Grade Model Serving
Serve models at any scale with one-click simplicity, with the option to leverage serverless compute.
Model Monitoring
Monitor model performance and how it affects business metrics in real time. Databricks delivers end-to-end visibility and lineage from models in production back to source data systems, helping analyze model and data quality across the full ML lifecycle, and pinpoint issues before they have a damaging impact.
Repos
Repos allows engineers to follow Git workflows in Databricks, enabling data teams to leverage automated CI/CD workflows and code portability.
Large Language Models
Databricks makes it simple to deploy, govern, query and monitor LLMs and integrate them into your workflows, and provides platform capabilities for augmenting (RAG) or fine-tuning LLMs using your own data, resulting in better domain performance. We also provide optimized tools to pretrain your own LLMs in days at 10x lower costs.
