Session

Ditch the Message Bus with Zerobus Ingest

Overview

ExperienceIn Person
TrackData Engineering & Streaming
IndustryManufacturing, Financial Services, Transportation
TechnologiesLakeflow
Skill LevelIntermediate
DOWNLOAD SESSION SLIDES

Streaming is evolving—quickly. Modern architectures are eliminating infrastructure sprawl in favor of a direct-to-lakehouse model, eliminating the need for intermediate message buses and the expense of maintaining them. Zerobus Ingest allows you to push data directly to the lakehouse with near real-time writes, high throughput, and support for thousands of concurrent clients.

 

In this demo-packed session, we’ll cover:

  • How Zerobus works and how to build production-ready pipelines—without the heavy ownership costs of DIY infrastructure.
  • The latest supported interfaces (e.g., REST, OTel, and Kafka-compatible APIs), as well as new features designed for massive scale (e.g., liquid clustering, JSON variant type support, etc.).
  • See how global innovators like Joby Aviation and Toyota Motor Corporation have embraced this shift to streamline their real-time architectures.

 

Join us and discover a direct ingestion framework to simplify your stack and accelerate your impact.

Session Speakers

Aleksandar Tomic

/Senior Staff Software Engineer
Databricks

Victoria Bukta

/Member of Product Staff
Databricks

Full Summary

Ditch the message bus: how Zerobus Ingest streams data straight to the lakehouse

Many streaming pipelines park a message bus like Kafka or Kinesis between producers and the lakehouse, using it only as a durable buffer before ingestion. That layer adds capacity planning, scaling complexity, and operational overhead that offers little value when the lakehouse is the sole consumer. For teams building real-time analytics on streaming data, the conversation argues for a simpler path. Organizations evaluating whether to consolidate their stack can find grounding in a complete guide to real-time conversational analytics on streaming data before committing to a broker-heavy architecture.

FAQ


Keep a broker when it powers microservice-to-microservice communication, multiple independent consumers, or stream processing that depends on broker semantics. Zerobus targets pipelines where the bus is only a buffer to Delta Lake tables.

No cluster or pipeline provisioning is needed. Create a Unity Catalog table, grant a service principal, and start pushing records via an SDK or REST. Zerobus is an always-on service available on the default Databricks endpoint.

Zerobus durably offloads events in roughly 200 milliseconds and makes data queryable in about five seconds. Clients receive offsets that confirm durability while streaming records continuously.

Zerobus validates every record against the target table schema and never creates or alters tables. Unknown fields or missing required columns cause explicit rejections. A rescue column is planned to persist non-conforming records for later triage.

Kafka producer protocol compatibility is planned, with support targeted so Kafka-native tools such as Kafka Connect and Debezium can be pointed at Zerobus with a configuration change. A demo showed Debezium writing MySQL CDC events directly into Delta Lake tables through Zerobus.