The Future of Data Orchestration: Asset-Based Orchestration
Data orchestration is a core component for any batch data processing platform and we’ve been using patterns that haven't changed since the 1980s. In this talk, I’ll be introducing a new pattern and way of thinking for data orchestration known as asset-based orchestration, with data freshness sensors to trigger pipelines. In this session we will demo this new pattern using popular tools of the modern data stack - dbt, Airbyte, Dagster, and Databricks.
- Lightning Talk
- In Person
- Data Engineering
- Media and Entertainment
- 20 min
Lead Engineer on the Dagster Project