HomepageData + AI Summit 2022 Logo
Watch on demand

ROAPI: Serve Not So Big Data Pipeline Outputs Online with Modern APIs

On Demand

Type

  • Session

Format

  • Virtual

Track

  • Data Engineering

Difficulty

  • Beginner

Duration

  • 0 min

Overview

ROAPI is a new data engineering tool written in pure Rust from the ground up. It provides a fully automated no code experience for spinning up online APIs for slow moving datasets produced from data pipelines. The core of its design can be boiled down to the followings:

- Query frontends to translate SQL, GraphQL and REST API queries into Datafusion query plans.
- Data layer to load datasets from a variety of sources and formats with automatic schema inference.
- Response encoding layer to serialize intermediate Arrow record batch into various formats requested by client.

Due to its in memory design, ROAPI operators can effectively scale the online API infinitely by throwing more instances behind a load balancer. Clients of ROAPI can choose and pick the most suitable query interface and response serialization format based on their needs.

In this talk, I will give a quick intro of what ROAPI is, it's design philosophy and how you can leverage it to make your data more accessible from within your organization.

Project home page and source code: https://roapi.github.io/docs, https://github.com/roapi/roapi.

Session Speakers

QP Hou

Engineer

Neuralink

See the best of Data+AI Summit

Watch on demand