Announcing the Public Preview of Zerobus Ingest

Zerobus Ingest, part of Lakeflow Connect, provides a streamlined way to push event data directly into the lakehouse, eliminating the single-sink message bus layer entirely. By reducing infrastructure, simplifying operations, and delivering near real-time ingestion at scale, Zerobus Ingest makes it easier than ever to unlock the value of your event data.

Traditionally, organizations use message buses like Kafka as a transport layer. While Kafka provides durability and low latency, it introduces a two-hop architecture: data is first written to Kafka, then pulled into the lakehouse with Spark Declarative Pipelines (SDP). This adds infrastructure, duplicates data, and demands specialized tools and expertise, all of which can make debugging harder. The outcome is higher costs, slower delivery, and more operational overhead—especially when the sole destination is the lakehouse.

With just a few lines of code, Zerobus Ingest enables you to create a stream and start pushing your data directly to your lakehouse.

Performance at scale, simplified architecture

Event data powers critical real-time analytics and AI applications. From clickstreams and telemetry to IoT sensor data, organizations depend on fast, reliable pipelines to make decisions in seconds, not hours. Zerobus Ingest is designed for performance at scale, delivering latency as low as five seconds, throughput up to 100 MB/sec per connection, and support for thousands of concurrent clients writing to the same table.

By eliminating data hops, redundant copies, and external message buses, Zerobus simplifies infrastructure and reduces operational overhead for event data ingestion. Fully unified with the Databricks Data Intelligence Platform, Zerobus writes natively to Delta tables, integrates with Unity Catalog for security and governance, and connects seamlessly with analytics and AI tools.

Zerobus Ingest Interfaces and SDKs:
Databricks supports an expanded set of interfaces to streamline ingestion and provides an SDK that allows you to build customized high-throughput applications using Zerobus Ingest. There is also support for standardized interfaces such as Open Telemetry to enable your existing applications to forward data with only a config change.

Zerobus Ingest SDKs and APIs support custom device integrations, enabling direct communication to managed Delta tables with minimal infrastructure. 

We are releasing the following interfaces:

• SDKs
  • SDK for Python (PuPr)
  • SDK for Java (PuPr)
  • SDK for Rust (PuPr)
• APIs
  • gRPC APIs  (PuPr) *
  • REST API (PrPr) *
  • OpenTelemetry APIs (PrPr) * 

* Contact your Databricks representative for more information.

These new interfaces enable support for your ever-growing throughput needs from source to sink, with no specialized infrastructure required. See the technical documentation for more details. 
 

Zerobus Ingest allows data producers to bypass the message bus and push events directly into managed Delta tables in your Lakehouse. 

Event Data Use Cases Across Industries

Zerobus Ingest is designed to handle diverse, high-volume event data at scale—across telemetry, clickstream, and IoT use cases supporting industries like manufacturing, telecommunications, e-commerce and retail, and IT and cybersecurity in near real-time. By connecting physical systems and digital environments directly to the lakehouse, Zerobus Ingest enables governed, high-performance analytics in near real-time. 

Manufacturing and IoT

Factories produce continuous streams of machine and sensor data for real-time and historical performance analysis to help improve efficiency and reduce unplanned downtime. They do this often under strict networking and integration constraints. With Zerobus Ingest and its direct-write SDK, teams can build custom forwarding agents to stream data straight into the lakehouse. Combined with Unity Catalog, this provides secure, low-latency ingestion with end-to-end governance from edge to analytics.

Joby Aviation is reimagining air travel with electric aircraft built from the ground up—manufacturing precision components and analyzing flight telemetry. Joby uses Zerobus Ingest to stream gigabytes of telemetry data per minute directly into their lakehouse. powering near real-time insights that support aircraft development, post-flight operational optimization, and manufacturing optimizations down to the machine level. Read the Joby Aviation case study. 

“With Zerobus Ingest, we’re pushing gigabytes of telemetry per minute to the lakehouse from our manufacturing sites… It’s not just that Zerobus is fast, it scales with us. We didn’t have to redesign anything to support more sites, more devices, or more data.” —Dominik Müller, Factory Systems Lead, Joby Aviation

Streamlined ingestion solution using custom Joby forwarding agents with Zerobus Ingest.

Telecommunications and IoT

For telecommunications organizations with globally distributed devices, real-time processing of millions of device signals per second is crucial for analytics, monitoring network performance, and optimizing customer experience

Databricks is partnering with Soracom, a global provider of IoT connectivity, to extend Zerobus Ingest to cellular, satellite, and LPWAN services worldwide. Together, they can help customers move data securely and reliably from any device to the lakehouse. 

“Our customers can now push high-fidelity data from devices around the world to the Databricks Data Intelligence Platform using our cellular, satellite and LPWAN services and accelerate time to insights for game-changing results.” — Kenta Yasukawa, CTO and Co-founder, Soracom, Inc.

Commerce, Retail and Clickstream

In retail and e-commerce, analyzing behavioral and clickstream data from websites, apps, and connected devices as it’s generated is key to driving near real-time personalization, A/B testing, and conversion optimization. Zerobus Ingest helps enable this use case by writing data directly to Delta tables, removing message bus dependencies and overhead infrastructure, and providing a single, analytics-ready source of truth.

IT and Cybersecurity

For security teams, fast access to telemetry and event data in near real-time is critical to reduce the latency between detection, model feedback, analysis, and response. Zerobus Ingest enables continuous, low-latency ingestion of logs and metrics to accelerate threat identification and adaptive model retraining. It can also enable near real-time fraud or anomaly detection by streaming behavioral events directly into your lakehouse, without the delays of traditional ETL or batch ingestion. 

Getting Started with Zerobus Ingest

As part of the Public Preview, we’re excited to see how customers across industries will use Zerobus to unlock new possibilities with the Databricks Data Intelligence Platform. To help you get started, we’ve also released several demos, a solutions accelerator, and step-by-step tutorials with sample data to help you experience the capabilities of Zerobus Ingest first-hand.  

In order to get Zerobus Ingest enabled on your workspace today, reach out to your Databricks account representative.

• Solution Accelerator: Digital Twins
  Deploy your own Digital Twin solution using native Databricks features (Zerobus Ingest, Lakeflow Declarative Pipelines, Lakebase) to reduce the costs and complexity of integrating a third party solution. Read blog | Github repo
• Tutorial: Using Databricks Apps as a Lambda with Zerobus to write to Delta
  A step-by-step tutorial on how to use the Databricks Direct Write App, a production-ready, modular FastAPI application designed for high-performance ingestion of structured data using Zerobus Ingest for Delta Lake tables. Key benefits include fast, streaming data transfer, and efficient and secure data loading. Read blog
• Tutorial: Streamline Data Ingestion with Zerobus Ingest: RabbitMQ to the Lakehouse
  Zerobus Ingest enables the creation of forwarding agents that easily subscribe to RabbitMQ messages, seamlessly pushing this data to Databricks and transforming it into Delta Lake tables for analytics. Read blog | Github repo
• Container Service: Zerobus-Station
  A scalable container service that provides a RESTful interface to Zerobus Ingest tables. 
  Read blog | Github repo