Feature and ML Platform Powered by Databricks
Overview
| Experience | In Person |
|---|---|
| Track | Artificial Intelligence & Agents |
| Industry | Enterprise Technology, Financial Services |
| Technologies | Lakeflow, Unity Catalog, Lakebase |
| Skill Level | Intermediate |
This presentation offers a comprehensive exploration of the Feature & Machine Learning Platform, focusing on its architectural design, essential components and seamless integration with Databricks. The platform harnesses Databricks for advanced feature engineering, structured streaming, declarative pipeline orchestration and end-to-end model training and serving, supporting both scheduled and event-driven workflows for maximum flexibility.
Key technologies highlighted include the DBX Feature Engineering SDK, Apache Spark Structured Streaming and Databricks Lakeflow jobs, all of which contribute to scalable, efficient data processing. Online feature computation is facilitated through the DBX Unity Catalog with Python UDFs, while Databricks Online Feature Stores powered by Lakebase enable real-time use cases. For batch feature consumption outside Databricks, managed Delta UniForm and Iceberg read-only feature tables provide robust solutions.