ROAPI: Serve Not So Big Data Pipeline Outputs Online with Modern APIs
- Data Engineering
- 0 min
ROAPI is a new data engineering tool written in pure Rust from the ground up. It provides a fully automated no code experience for spinning up online APIs for slow moving datasets produced from data pipelines. The core of its design can be boiled down to the followings:
- Query frontends to translate SQL, GraphQL and REST API queries into Datafusion query plans.
- Data layer to load datasets from a variety of sources and formats with automatic schema inference.
- Response encoding layer to serialize intermediate Arrow record batch into various formats requested by client.
Due to its in memory design, ROAPI operators can effectively scale the online API infinitely by throwing more instances behind a load balancer. Clients of ROAPI can choose and pick the most suitable query interface and response serialization format based on their needs.
In this talk, I will give a quick intro of what ROAPI is, it's design philosophy and how you can leverage it to make your data more accessible from within your organization.
Project home page and source code: https://roapi.github.io/docs, https://github.com/roapi/roapi.