As the worlds of data, analytics and AI become increasingly interconnected, there’s a growing need for a platform to unify data sources, use cases and workloads. The data lakehouse removes the barriers that separate business analysts from data scientists, structured data from real-time streaming and business intelligence from AI. This opening keynote explores how the lakehouse’s simple, open, multicloud approach simplifies data management, unifies data teams and productionizes more initiatives.

Using a demo-driven approach, this keynote will clearly delineate how the major data workloads of data engineering, data warehousing, data science and machine learning are orchestrated within the architecture of the Databricks Lakehouse Platform.

• The Case for Lakehouse, Breaking down the data and governance silos
• The Case for Lakehouse, Customer Story
• Data Engineering on the Lakehouse
• Data Warehousing on the Lakehouse
• Machine Learning on the Lakehouse

Data engineers have the difficult task of cleansing complex, diverse data, and transforming it into a usable source to drive data analytics, data science, and machine learning. They need to know the data infrastructure platform in depth, build complex queries in various languages and stitch them together for production. Join this talk to learn how Delta Live Tables (DLT) simplifies the complexity of data transformation and ETL. DLT is the first ETL framework to use modern software engineering practices to deliver reliable and trusted data pipelines at any scale. Discover how analysts and data engineers can innovate rapidly with simple pipeline development and maintenance, how to remove operational complexity by automating administrative tasks and gaining visibility into pipeline operations, how built-in quality controls and monitoring ensure accurate BI, data science, and ML, and how simplified batch and streaming can be implemented with self-optimizing and auto-scaling data pipelines.

To make the lakehouse a reality, the query engine needs to support both structured and unstructured data, while providing the performance of a data warehouse and the scalability of data lakes. In this session, learn how Databricks’ next generation vectorized engine Photon outperforms existing data warehouses in SQL workloads and implements a more general execution framework for efficient processing of data with support of the Apache Spark™ API.

As companies roll out ML pervasively, operational concerns become the primary source of complexity.  Machine Learning Operations (MLOps) has emerged as a practice to manage this complexity.  At Databricks, we see firsthand how customers develop their MLOps approaches across a huge variety of teams and businesses.

In this session, we will share how Databricks uniquely solves this by unifying the key aspects of MLOps, namely DataOps, ModelsOps and DevOps, on a unified platform through the Lakehouse, enabling faster and more reliable production ML . We will show how your organization can build robust MLOps practices incrementally. and unpack general principles which can guide your organization’s decisions for MLOps, presenting the most common target architectures we observe across customers.

Customers around the world are experiencing tremendous success migrating from legacy on-premises Hadoop architectures to a modern Databricks Lakehouse in the cloud. At Databricks, we have formulated a migration methodology that helps customers sail through this migration journey with ease. In this talk, we will touch upon some of the key elements that minimize risks and simplify the process of migrating to Databricks, and will walk through some of the customer journeys and use cases.

Modern data assets take many forms: not just files or tables, but dashboards, ML models, and unstructured data like video and images, all of which cannot be governed and managed by legacy data governance solutions. Join this session to learn how data teams can use Unity Catalog to centrally manage all data and AI assets with a common governance model based on familiar ANSI SQL, ensuring much better native performance and security. Built-in automated data lineage provides end-to-end visibility into how data flows from source to consumption, so that organizations can identify and diagnose the impact of data changes. Unity Catalog delivers the flexibility to leverage existing data catalogs and solutions and establish a future-proof, centralized governance without expensive migration costs. It also creates detailed audit reports for data compliance and security, while ensuring data teams can quickly discover and reference data for BI, analytics, and ML workloads, accelerating time to value.

Everyone wants to reduce the time it takes to turn data into actionable information. This often involves integrating several data tools in what has become known as the modern data stack (MDS). But, most approaches have only focused on half the problem by rooting the MDS in the data warehouse. A true MDS should solve all modern problems, and this means tackling AI and streaming in addition to reporting and BI. In this deep dive demo session, we show you how easy it is to integrate the Databricks Lakehouse Platform into your modern data stack to connect all your data tools across SQL, AI/ML, and streaming, and discover new methods to unlock insights faster.