delivery

track

type

level

industry

SVP, Data and Analytics

Mitchell Kearney

A Data and Analytics Ecosystem, Purpose-Built for Healthcare

Use the Databricks Statement Execution API to create automation in the programming language of your choice, for example, by receiving a notification upon completion of a long-running query, to trigger hardware using the Viam SDK in a serverless function. This session is a gentle introduction to using the Databricks API, simplifying access to your data and making it easier to build data applications tailored to your use cases.

Head of Developer Relations

Joyce Lin

A Guide to Building Smart Industrial Automations with the Databricks API

As part of their global AI strategy, companies want to ensure that they are at the forefront in developing and implementing cutting-edge technology. A large chunk of that AI strategy is to provide hundreds and thousands of employees with the tech stack to build and/or consume GenAI applications with proper governance and control. But what are the components of that state-of-art architecture? In this showcase, we will review architecture that involves Databricks, Dataiku LLM Mesh, Nvidia, and CSPs. We will walk through a RAG based application for a practical use case that will leverage the architecture discussed with pre/post processing of data. We will discuss how to operationalize the use case with proper governance and controls required in an enterprise grade application. This showcase will be able to provide a proof point of the modern day architecture for creating a platform for GenAI developers in your organization. And did I say we will do this with very few lines of code if any!

RVP - Global Partner enablement

Anjaney Shrivastava

A How to Guide to Design an Enterprise GenAI Platform 

In a traditional data service, providers first develop their product in one platform and load it onto another platform to download or extract the data, which has to be ingested, transformed, and loaded into another platform. Because it offers various tools in one place, Kythera Labs built our data-transforming platform Wayfinder on the Databricks Data Intelligence Platform. Wayfinder leverages many of Databricks’ features to create our data products, distribute our data, and enable our customers to use our data products cost-effectively and centrally. This session highlights the many features of Wayfinder, which gives our customers the full power of a data intelligence platform, including Delta Sharing, Unity Catalog, ELT, Orchestration, Warehousing, and GenAI.

CTO and Co-Founder

Matt Jocker

Solutions Architect Manager

Josue Bogran Micher

A Lean Data Factory: The Power of a Data Intelligence Platform

This session is repeated. Is using dimensional modeling and star schemas, not a good architecture for data marts or data products in Databricks? Think again. Star schemas still make the best data model for your gold layer. The world is turning data-driven, and “everybody” is doing analytics. Many newcomers to analytics are led to think analytics is only about making data available. Data marts with so-called wide tables are popping up in gold layers, providing minimal analytical value, and are a nightmare when it comes to data governance and data quality. This session will give you an overview of why star schemas are the winning data model for your gold layer. The differences between relational and columnar databases mean that you can simplify some of the physical implementation of your star schemas. Therefore, we will also give you some design techniques that you should consider in Databricks and touch on some best practices for your physical implementation.

Truls Bergersen

A Modern Approach to Dimensional Modeling - In a Columnar Database (repeat)

A Modern Approach to Dimensional Modeling - In a Columnar Database (repeated)

This session introduces Databricks' latest advancements in real-time inference, showcasing how its cutting-edge feature and function serving capabilities are revolutionizing the way AI models and data can be leveraged for instantaneous decision-making.Attendees will dive deep into the mechanics of Databricks' real-time inference ecosystem. You'll discover practical strategies for deploying your AI models as robust, externally facing APIs, enabling seamless integration with a wide array of applications and services.Key takeaways include:<ul>	<li>An in-depth understanding of Databricks' new real-time inference features and how they simplify the deployment of AI models.</li>	<li>Step-by-step guidance on setting up and managing feature and function serving to enable real-time data processing and insights.</li>	<li>Insights into how real-time inference can transform your organization's approach to data analysis, decision-making, and customer engagement.</li></ul>

Sr. Director, Product Management

Craig Wiley

Technical Marketing

Nicolas Pelaez

A Practical Introduction to Machine Learning with Databricks Mosaic AI

Generative AI and LLMs can bring transformative value to businesses. LLM application archetypes are becoming increasingly clear, and the technology ecosystem is maturing. However, most companies struggle to build and efficiently operationalize LLM applications. LLMOps, as a discipline, is emerging as a solution to overcome the challenges throughout the LLM development lifecycle. This session will explore the development of LLMOps from a practitioner’s perspective, highlighting how it addresses key concerns like cost optimization, risk mitigation, and improved collaboration between stakeholders. Real-world case studies and lessons learned will be shared, offering practical insights for fellow practitioners embarking on the exciting and challenging world of LLMOps.

Sr. Engineering Manager

Hien Luu

A Practitioner's Perspective on LLMOps

For the past decade, the prevailing approaches to scaling ML across organizations and infrastructure have centered on model and pipeline portability. Essentially, try to produce a unified compiler to drop your model onto any hardware, and try to produce a unified pipeline format to drop your workflow onto any compute. Ten years later, these approaches haven't panned out, and as a result, enterprise ML is alarmingly fragmented. We propose an entirely new approach: to leave the ML methods (algorithms, models, pipelines, utilities, etc.) in place on the hardware or storage they were defined to live on and make them instantly sharable and accessible from anywhere. Think of it as Google Docs instead of emailing files. We propose a set of open source PyTorch-like APIs that make creating and sharing such shared apps and services intuitive, production-grade, and scalable and imbue the system with a deep agnosticism to the underlying infra or existing ML tooling already in place.

Software Engineer

Rohin Bhasin

Caroline Chen

A PyTorch Approach to ML Infrastructure

The objective of this presentation is to showcase how Doordash and Databricks teams came together to accelerate the pace of Databricks adoption for ML and streaming usecases and how this  helped accelerate the adoption of Databricks for workloads that perform more optimally with Delta and Spark compute. However, adoption, in general, can be time-consuming and operationally intensive. Operational efficiency and enablement through automation is a critical factor in making the adoption of any new platform successful.The focus of this topic will primarily be around tooling and Databricks native features that can help audiences who are on a similar journey. Databricks Accelerators that will be covered in this discussion include Databricks Migration Diagnostics Tool, Delta Lake object builder, Apache Spark™ SQL translator, Delta Lake validation and reconciliation tool, and Databricks Airflow dag migration tool.

Engineering Manager

Harsha Reddy

Aydar Akhmetzyanov

Accelerating Adoption of Datalake for Streaming/ML Use Cases

Financial services firms are at the forefront of harnessing the power of data to drive innovation, improve customer experiences, and enhance operational efficiency. This panel discussion aims to explore the multifaceted benefits and strategic approaches to leveraging the Databricks Marketplace for fostering data-driven collaborations. Experts from leading financial institutions like RBC Capital Markets will delve into practical insights on overcoming common challenges in data sharing, such as data privacy, security concerns, and interoperability issues. Participants will share vision and learnings of how integrating with the Databricks Data Intelligence Platform has not only streamlined data workflows but also unlocked innovative analytics and AI capabilities, driving significant business value. Whether you’re a data provider looking to monetize your assets or a consumer aiming to enrich your analytical models, this session will equip you with the knowledge and strategies to effectively navigate the data ecosystem within the Databricks Marketplace, setting a new standard for collaboration in financial services.

Sr. Director, Data Partnerships

Jay Bhankharia

Head of Alternative Data and AI

Sandeep Earayil

Data Product Specialist

Adam Longenecker

VP, Head of Distribution Platforms

John Schirripa

Accelerating Financial Innovation: Unlock Insights with Databricks Marketplace

In today's GenAI-crazed world, companies across all industries and segments have multiple IT-driven GenAI POCs being developed and evaluated in parallel - the clients that are moving use cases to production the fastest and disrupting marketplaces are collaborating with their business counterparts to drive business value with the proper use cases. During this talk, Lovelytics, one of Databricks' premier consulting partners, will share lessons from their clients that are moving the fastest to unlock business value with GenAI on Databricks. Learn how their clients are moving the fastest to unlock business value with GenAI on Databricks. By understanding the value of pre-trained transformers, Data and AI leaders can work with their business counterparts to target the proper use cases.

Managing Director of AI

Sudhir Gajre

Accelerating GenAI with Lovelytics: The Power of Pre-Trained Transformers

Artificial intelligence is fast becoming a generation-defining technological breakthrough. Hear how technology leaders from JP Morgan Payments Data team are putting AI to use, from encouraging agile technical experimentation to driving real business outcomes. You’ll also learn how to plan your own AI investments by evaluating emerging use cases and shifting organizational culture. See how JP Morgan Payments Engineering have built a self-service innovation factory with Databricks and MLflow for Data Scientists, Data Analysts and Data Engineering (#DataSciencefirst, #DataAnalystFirst, #DataEngineerFirst)

Chief Engineer

Oleg Uris

Executive Director

Satya Vandrangi

Accelerating J.P. Morgan Data Product Development in the Age of AI

vLLM is an open-source highly performant engine for LLM inference and serving developed at UC Berkeley. vLLM has been widely adopted across the industry, with 12K+ GitHub stars and 150+ contributors worldwide. Since its initial release, the vLLM team has improved performance by more than 10x. This session will cover various topics in LLM inference performance, including paged attention and continuous batching. Then, we will focus on new innovations we’ve made to vLLM and the technical challenges behind them, including: Speculative Decoding, Prefix Caching, Disaggregated Prefill, and multi-accelerator support. The session will conclude with industry case studies of vLLM and future roadmap plans. Takeaways: <ul>	<li>vLLM is an open source engine for LLM inference and serving, providing state-of-the-art performance and an accelerator-agnostic design.</li>	<li>In focusing on production-readiness and extensibility, vLLM’s design choices have led to new system insights and rapid community adoption.</li></ul>

PhD student

Zhuohan Li

Cade Daniel

Accelerating LLM Inference with vLLM

This session will showcase how our Financial/Insurance company implemented a Retrieval Augmented Generation (RAG) system to enhance customer service efficiency. We'll provide an overview of our RAG architecture and how we used Databricks to build a robust data pipeline for indexing content and collecting user feedback. We'll explore innovations like multi-stage content chunking, advanced search retrieval techniques, and an evaluation framework for optimization. Our feedback loop using Databricks workflow improves the RAG implementation. We'll also highlight our unique orchestration layer that accelerates Generative AI use cases. By the end, you'll understand how Generative AI can transform customer service operations with strategies for efficiency and automation. These learnings apply to any customer service organization striving for operational greatness.

Principal Engineer

Peter Landis

Lead Data Engineer

Gen Li

Accelerating Operational Excellence with Generative AI

Unity Catalog is the bridge to unlock advanced analytics, GenAI use cases, and productivity gains on the Databricks Data Intelligence platform. During ths talk, you’ll understand how Tredence UnityGo! removes the complexities of manual migrations and appreciate how simple solutions are often the most powerful.

Senior Director, Partner Development

David LeGrand

Director, Business Value Acceleration

Kathleen Wong

Accelerating Unity Catalog ROI Through Meta Data Transformation

AccuWeather utilizes its rich and quality-assured weather data in a proprietary algorithm that interprets forecasted weather into an impact value tailored to users’ data. By leveraging the most comprehensive weather datasets on the Databricks Marketplace, AccuWeather enables businesses to uncover powerful relationships between weather and operational metrics. With incomparable expertise and advanced AI capabilities, AccuWeather’s patented methodology translates weather information into actionable insights and delivers Superior Accuracy™. This approach helps users identify significant weather impacts on their business, facilitating proactive, data-driven decisions, and mitigating weather-risk. Leverage the AccuWeather Data Suite on the Databricks Data Intelligence Platform to understand how weather impacts your bottom line.

Manager, Data Science

Timothy Loftus

AccuWeather®: Turn Risky Weather Obstacles into Business Opportunities

In this technical talk, attendees will discover how Intuit has leveraged open-source MLFlow to improve the performance of traditional ML models and LLM powered applications through effective tracking and management of machine learning experiments. Highlights of the talk include Intuit's customizations for achieving reproducibility of experiments, model lineage tracking, and streamlined model review workflow through MLFlow integration with their MLPlatform experience.

Sr. Staff Software Engineer

Manas Mukherjee

Advanced Experiment Tracking for LLM-Powered applications with Customized Open-Source MLFLow

Join our speakers from Shell Energy as they discuss the challenges and solutions in implementing a data strategy and lakehouse in a large, diverse organization. They’ll share their experiences, including initial hurdles in data strategy and governance, and how they used Unity Catalog and a business-owned data product approach to overcome them. They’ll delve into the concept of Data Mesh, discussing the roles of the product team and the customer, and provide real-life examples. They’ll also share insights on using analytics, PowerBI, ML models, and AI for Data Governance. This session is for professionals grappling with data strategy and governance, offering insights on aligning goals, identifying use cases, team organization and scaling operations. Attendees will gain a deeper understanding of achieving success with data, analytics, and AI. The session, aimed at an intermediate level, will include code snippets and working code demonstrations.

Product Manager

John O'Brien

AI and the Lakehouse: Shell’s Journey Towards Effective Data Governance

Come learn about the latest Salesforce and Tableau innovations and how it can help drive more value from your existing technology investments in Databricks. We will showcase how Delta Sharing is used for secure access when sharing with external parties for visualization, how you can leverage your existing ML models for predictive and hyper-personalized customer engagements in the Salesforce 360 platform, and also how you can simplify your data pipeline with our new zero-copy integration.

Principal Technical Architect

Tim Cook

AI Driven Engagements with the Salesforce and Databricks Partnership

In today's data-driven world, organizations are constantly seeking ways to leverage artificial intelligence (AI) to unlock insights and drive innovation. However, integrating AI into existing data workflows can be challenging, often requiring specialized knowledge and complex infrastructure. To address this challenge, Databricks has introduced AI Functions, a powerful feature that enables users to access large language models (LLMs) directly from SQL, simplifying the process of incorporating AI into data workflows. In this session, we will explore the capabilities of AI Functions and how they can be used to enhance data analysis in Databricks SQL. We will demonstrate how to use AI Functions to perform tasks such as natural language query generation, data documentation, and custom logic creation. We will also discuss best practices for using AI Functions, including managing access control and ensuring data privacy.

Jeremy Lewallen

AI Functions: Using AI in SQL

Cybersecurity incidents are costly, and using an endpoint detection and response (EDR) solution enables the detection of cybersecurity incidents as quickly as possible. To effectively detect cybersecurity incidences requires the collection of millions of data points and the storing and querying of endpoints data presents considerable engineering challenges without creating internal data silos. Databricks tooling enabled us to break down our data silos and iteratively improve our EDR pipeline to ingest data faster and reduce querying latency by more than 20% while reducing costs by more than 30%. In this session, we will share the journey, lessons learned, and the future for collecting, storing, governing, and sharing data from endpoints in Databricks. The result of building EDR using Databricks helped us accelerate the deployment of our data platform and power our cybersecurity co-pilot.

Distinguished Data Architect

Justin Lai

Director Product Management

Robert Lombardi

Sr. Solutions Architect

Digan Parikh

AI-Powered EDR: Streamlining Blackberry Cybersecurity with Databricks

While many companies are already experimenting with AI or ML, not all have been able to capitalize on their potential fully. This is where our Accounts Receivable Forecasting solution comes in. Our solution utilizes CICD pipelines to automate the deployment of Azure cloud infrastructure, ELT and ML notebooks, and an ML monitoring dashboard. With this solution, companies without any prior cloud infrastructure experience can fully productionize their AR forecasting model in just a matter of days. In this session, we will: <ul>	<li>Introduce AR forecasting (and why it is relevant for every company)</li>	<li>Introduce the broader concepts of Terraform, CICD, and MLflow (in Databricks)</li></ul> Explain how we used: <ul>	<li>Terraform to set up the Infrastructure in the Azure Cloud</li>	<li>CICD pipelines to orchestrate the deployment of the infrastructure as well as the ML code</li>	<li>MLflow to support automated deployment and scheduling of the ML notebooks</li>	<li>A dashboard to monitor the ML models</li></ul>

Co-founder

Floriant Sturm

Data Scientist

Julie Vanackere

AI-Powered FinTech: AR Forecasting with Databricks & MLFlow

In this session, we will present our unique use case to provide a model serving for an internal pricing analytics application that triggers thousands of models in a single click and expects to receive a response in near real-time. We will provide an in-depth overview of how we have achieved this complex requirement through our in-house developed model serving framework and Databricks serverless compute. Attendees will leave with an idea of a model serving framework, how to make it more generic, and techniques they can use to employ the models and how to use Databricks serverless compute option effectively to achieve the performance at scale and keep the costs at a minimum.

Senior AI Engineer

Amit Adiraju Narashimha

Sr. Director Data Science

Diana Adam

Albertsons - Near Realtime Model Serving with Databricks Serverless 

Have Delta Lake and Data Engineering questions? Join this Delta Lake Ask Us Anything with Delta Lake maintainers and contributors!

Developer

R Tyler Croy

Distinguished Software Engineer

Scott Haines

Bart Samwel

Sr. Staff Developer Advocate

Denny Lee

Staff Software Engineer

Tathagata Das

Communications & Marketing Manager

Carly Akerly

AMA - Delta Lake

Have Generative AI questions? Join this GenAI Ask Us Anything with the Databricks Mosaic Research team!

Chief Scientist - Neural Networks

Jonathan Frankle

Sr. Research Scientist

Cody Blakeney

Research Scientist

Jasmine Collins

Sasha Doubov

Senior Research Scientist

Alexander Trott

AMA - Generative AI

Have Data Science, ML, and LLM questions? Join this Productionizing ML and LLMs Ask Us Anything with Databricks product managers, engineers, and data scientists.

Staff Data Scientist

Yinxi Zhang

Sr Staff Product Manager

Eric Peter

Director, Machine Learning Practice

Brooke Wenig

CME Lead Solution Architect

Layla Yang

Sr Director, Research

Xing Chen

AMA - Productionizing ML and LLMs

Have Apache Spark™ and Data Engineering questions? Join this Apache Spark Ask Us Anything with Apache Spark maintainers and contributors!

Hyukjin Kwon

Sr. Software Engineer

Allison Wang

Martin Grund

Engineering Director

Xiao Li

Wenchen Fan

AMA - Spark

This session will delve into the latest advancements in Apache Spark™ 3.5, highlighting its pivotal role in pushing the boundaries of big data processing and AI. We'll discuss Spark Connect's role in enhancing accessibility, DeepSpeed's integration for AI efficiency, and performance optimizations. Additionally, we'll delve into the new PySpark and SQL features, including built-in functions for array manipulation, SQL IDENTIFIER clause enhancements, expanded API support, and Arrow-optimized Python UDFs, highlighting their impact on building scalable, efficient, and robust data-driven applications.

Daniel Tenedorio

An In Depth Look at the New Features of Apache Spark 3.5

Apache Hudi, Delta Lake and Iceberg have emerged as leading open source projects, providing decoupled storage with primitives for transaction and metadata layers, known as table formats in cloud storage. When data is written to a distributed file system, all these formats store data in open columnar formats like Parquet, with metadata for schema, commit history, partitions and column stats. Choosing a table format is challenging; each project offers unique features. Choosing a table format is challenging as each project offers unique features. Enter XTable—an OSS project that provides omnidirectional interoperability between table formats. XTable doesn't introduce a new format but provides abstractions for translating metadata. This enables writing data in any format and converting it to targets consumable by different compute engines. This session showcases XTable's solution to the challenge of format selection and interoperability in lakehouse workloads.

Head of Product

Kyle Weller

Staff Developer Advocate

Dipankar Mazumdar

Apache XTable (incubating): Interoperability Among Lakehouse Table Formats

This session is repeated. We delve into practical applications and examples of Stanford DSPy, a programming model designed to enhance the development and optimization of language model (LM) pipelines. Unlike traditional LM pipelines that rely on rigid, hard-coded prompt templates, DSPy utilizes text transformation graphs and parameterized modules to create adaptive, self-improving pipelines. We will explore case studies demonstrating how DSPy programs can efficiently solve complex tasks such as complex question answering. By leveraging DSPy’s ability to compile and optimize pipelines for metrics, attendees will learn how even a few lines of code can significantly boost performance, allowing both large and small language models to achieve superior results with minimal effort.

Researcher

Thomas Joshi

Applications of Stanford DSPy for Self-Improving Language Model Pipelines (repeat)

Applications of Stanford DSPy for Self-Improving Language Model Pipelines (repeated)

Get more out of your data by creating data apps for your non-technical users to make data driven decisions. In this session, we cover:<ul>	<li>Building internal apps for your organization for data visualization, data entry, and AI.</li>	<li>Creating apps rapidly using Python and frameworks such as Dash, Gradio, and Streamlit</li>	<li>Accessing your data and models using SQL, Spark, and Model Serving.</li>	<li>The fast way to secure and govern your apps</li>	<li>Running your apps in production</li>	<li>What’s new for app developers on Databricks</li></ul>

Andre Furlan Bueno

Director of Product

Tarek Madkour

Apps on Databricks: Build Data Applications on Databricks in 20 Minutes!

We are rebuilding our data lakehouse from the ground up - a greenfield re-design based on everything we have learned from the last 5 years of working with Databricks. In this talk, we’ll give an architectural overview of how we’re setting up this new lake, covering topics like: <ul>	<li>How we are laying out our Databricks accounts/workspaces and AWS accounts to create the concept of “environments”</li>	<li>Our “workbench environment” concept that separates insights work from production pipelines</li>	<li>The benefit we get from doubling down on Unity Catalog, Delta Lake, and Managed Tables</li>	<li>Creation and enforcement of a consistent information architecture</li>	<li>Making our data lakehouse declarative, and keeping human users out of the production environment</li>	<li>Supporting and governing machine learning workloads</li></ul>

Principal Data Platform Engineer

Perry Stephenson

Senior Data Platform Engineer

Chen Zhou

Architectural Overview of Atlassian's Next-Generation Data Lakehouse

GPUs are well-known as accelerators for DL and ML workloads. This session will describe how GPUs can accelerate batch ETL operations. We will review the CPU and GPU architectures, including an overview of their memory subsystems. We provide a roofline analysis for individual database operations like joins, aggregations, and data compression. We discuss why these operations are well suited for GPU acceleration and can achieve up to an order of magnitude speedup. Using industry-standard benchmark queries, we demonstrate full end-to-end SQL query acceleration using GPUs in a prototype query engine. We compare the results to existing CPU solutions. Finally, we will review the performance of the same queries at a 3TB scale with the RAPIDS Accelerator for Apache Spark™, a plugin to Apache Spark™ that enables GPU acceleration with no code change.

Distinguished System Software Engineer

Jason Lowe

Senior AI Developer Technology Manager

Nikolay Sakharnykh

Architecture Analysis for ETL Processing: CPU vs GPU

Data teams today are faced with the need to rapidly onboard an increasing number of AI and data engineering projects while simultaneously accelerating time to value. Join us for an insightful session on AT&T's evolution on the Databricks Platform as they scale beyond their initial use cases to enable several lines of businesses. From automated pipelining with Delta Live Tables to serverless Databricks SQL warehouses and AI/ML use cases, Databricks has empowered the streamlining and acceleration of new data products at AT&T. However, it can be challenging to meet complex security and connectivity requirements when workloads are not deployed on your own network. This session describes how AT&T meets stringent security and regulatory requirements while adopting the Databricks serverless platform, starting with serverless SQL warehouses. We also describe the latest enhancements that Databricks Platform team has built in partnership with AT&T.

AT&T's Journey Towards a Serverless Data Intelligence Platform

Explore the Sessions

SESSIONS