Announcing General Availability of Zerobus Ingest, part of Lakeflow Connect

As organizations scale real-time operational intelligence, traditional streaming architectures have become costly bottlenecks. Managing message buses like Kafka and handling schema registries and connector frameworks creates a significant “complexity tax” that diverts high-value engineering resources from strategic business initiatives. Meanwhile, duplicate storage inflates cloud bills and multi-hop architectures delay critical insights. Finally, data in transit often sits outside centralized governance frameworks, creating compliance risks and lineage blind spots. 

Introducing Zerobus Ingest: Near Real-Time Streaming to the Lakehouse

Today, we’re excited to announce the General Availability of Zerobus Ingest, part of Lakeflow Connect. Zerobus Ingest is a fully managed, serverless service that streams data directly into governed Delta tables, removing intermediate layers to deliver a simplified, high-performance architecture. 

By enabling data to flow directly from producers to the lakehouse, Zerobus Ingest slashes costs and eliminates tool sprawl. It also delivers high performance at scale, supporting thousands of concurrent connections and achieving over 10GB/second of aggregate throughput to a table in under 5 seconds. 

The Single-Sink Advantage: Simplified Architecture for Major Cost Reduction

Traditional message buses like Kafka were designed as multi-sink architectures: universal hubs that route data to dozens of independent consumers. However, this flexibility can come at a steep cost when your sole destination is the lakehouse. Zerobus Ingest uses a fundamentally different approach, with a single-sink architecture optimized for a single job: pushing data directly to the lakehouse. 

This architectural choice eliminates complexity and drastically reduces cost:

• No brokers to scale as your data volume grows
• No partitions to tune for optimal performance
• No consumer groups to monitor and debug
• No cluster upgrades to plan and execute
• No specialized expertise, such as Kafka, is required on your team  

With Zerobus Ingest, there’s a single, managed Databricks endpoint. Create your table in Unity Catalog, start writing data with the API or SDK, and you’re done. That’s it, nothing else to set up. The serverless architecture automatically scales up to support gigabytes-per-second ingestion without any configuration changes. 

Zerobus Ingest allows data producers to bypass the message bus and push events directly into managed Delta tables in your Lakehouse. 

Zerobus Ingest simplifies the traditional streaming architecture from five managed systems down to two components, eliminating multiple failure points, reducing operational overhead and removing the need for specialized expertise.

• Traditional architecture: Source systems → Message Bus (Kafka Cluster) with a Schema Registry → Connectors → Lakehouse
• Zerobus Ingest architecture: Source systems → Zerobus Ingest → Lakehouse

By eliminating the intermediate message bus, you remove two major cost centers: the compute and storage for the bus itself, and the dedicated engineering time needed to manage it. Zerobus Ingest offers ingestion at a fraction of the cost per gigabyte compared to running and maintaining a self-managed Kafka cluster.

Zerobus Ingest offers ingestion at a fraction of the cost per gigabyte compared to running and maintaining a self-managed Kafka cluster.

Learn more about how Zerobus works in this deep dive Databricks Community blog or in the documentation.

Supported Interfaces and Native Integration

Developers can integrate via gRPC and REST APIs, or use language-specific SDKs. Zerobus Ingest provides a broad set of push-based interfaces for industry-specific integrations, making it a flexible, single tool that simplifies ingestion.

• gRPC API: Recommended for high-performance applications requiring the lowest latency and highest throughput.
• REST API (Beta): Ideal for webhooks, serverless functions, and languages where gRPC support may be limited.
• SDKs: Production-ready libraries for Python, Java, Rust, Go, and TypeScript simplify authentication and batching logic utilizing gRPC.
• Open Telemetry (Beta): Bring your operational logs, metrics, and traces into the Lakehouse for long-term historical analysis with just a config change. Learn more about the Open Telemetry ecosystem here.

Learn more about the differences between REST and gRPC.

Also, since every write is governed by Unity Catalog, you get automatic lineage tracking and fine-grained access control from the moment data is created—ensuring your streaming data has unified governance with the rest of your lakehouse. 

Driving Customer Breakthroughs: Exponentially Faster Insights at Scale 

Real-Time Manufacturing Monitoring for Toyota Motor Corporation

Toyota sought a unified solution to instantly process telemetry from thousands of factory devices, without the latency and complexity of traditional IoT architectures. 

"Zerobus Ingest allows us to detect overheating factory conditions in minutes rather than hours, directly supporting our carbon-neutrality strategy and operational efficiency. But it's not just about tracking temperature telemetry; having Zerobus Ingest as an additional data ingestion option gives us the ability to collect diverse factory data in near real-time and trigger immediate countermeasures that have transformed our operations.” —Kento Izumi, General Manager, Digital Transformation Promotion Division, Toyota Motor Corporation

Instead of stitching together multiple cloud services, Toyota uses Zerobus Ingest, integrated with global IoT connectivity from Soracom, to mitigate the high maintenance costs of real-time operations, transform its manufacturing operations, and support its sustainability goals.

IoT Data Pipeline Architecture: From Edge to Analytics Platform with Zerobus Ingest and Soracom Beam

Izumi also explained that they are able to accelerate their operational efficiency, “When combined with 'vista,' our unified data and AI platform powered by Databricks, we aren't just collecting data faster; we are optimizing our data operations."

Joby Aviation: Accelerating Flight Performance Analysis From Days to Minutes

An early adopter of Zerobus Ingest, Joby Aviation streams gigabytes of aircraft telemetry every minute directly to the lakehouse, enabling their engineering teams to analyze flight performance in near real-time. Read the Joby Aviation case study.

"Zerobus Ingest reduced our telemetry resolution latency from days to minutes. This allows our engineering teams to analyze flight performance in near real-time and accelerate our mission to transform transportation." —Dominik Müller, Factory Systems Lead, Joby Aviation

Powering Industry Use Cases

Traditional infrastructure slows down real-time operations. By removing the complexity of intermediate message buses, Zerobus Ingest creates a direct, sub-5-second path to value across industries.

Accelerate your digital transformation by pushing data from any source across industries directly to your lakehouse.

Manufacturing: Maximize factory floor efficiency. Use the Zerobus Ingest SDKs to build custom forwarding agents that stream massive sensor volumes to the Lakehouse. This optimizes machine performance by eliminating heavy network infrastructure overhead.

Telecommunications and IoT: Monitor global networks at scale. Deployed at the edge, Zerobus Ingest pipes telemetry from your network to the lakehouse to track your network load in near real-time. Our partnership with Soracom expands integration with secure, reliable global IoT data ingestion through cellular, satellite, and LPWAN networks.

IT and Cybersecurity: Identify threats without the ETL delay. Bypass complex pipelines by streaming logs and behavioral events directly to the Lakehouse. This enables threat detection within seconds, adaptive model retraining, and faster incident response.

Commerce and Clickstream: Personalize experiences in near real-time. Capture high-volume clickstream data from apps and devices with minimal infrastructure overhead. This enables instant data availability to power personalization engines, A/B testing, and conversion optimization.

Availability

Zerobus Ingest is now Generally Available on AWS, Microsoft Azure, and Google Cloud platform support coming soon. Pricing is volume-based under the Lakeflow Jobs Serverless SKU.

As part of the GA launch, we are introducing a 6-month promotional pricing period. Learn more at the Lakeflow Connect pricing page. 

Getting Started with Zerobus Ingest

Ready to eliminate streaming infrastructure complexity? With just a few lines of code, you can begin streaming data directly to your Unity Catalog-governed tables, ensuring your data is ready the moment it arrives to help deliver insights.

Check out the following Zerobus Ingest resources to get started today:

• Try Zerobus Ingest Now: Access the documentation and quickstart guides.
• Take Product Tour: Navigate through Zerobus Ingest and learn how to get started ingesting data.
• Build an End-to-End Application: A real-time sailing simulator tracks a fleet of sailboats using Python SDK and the REST API, with Databricks Apps and Databricks Asset Bundles. Read the blog.
• Build a Digital Twins Solution: Learn how to maximize operational efficiency, accelerate real-time insight and predictive maintenance with Databricks Apps and Lakebase. Read the blog.