Rewind, Replay, Recover: Building Operational Resilience at NAB With Spark Declarative Pipelines

Discover how National Australia Bank (NAB) scales production streaming with Spark Declarative Pipelines (SDP), using Streaming Tables and Materialized Views to enable rewind and replay — restoring pipelines to a consistent state, applying fixes safely and reprocessing only affected data. Explore how this approach reduces brittle manual interventions and avoids full refreshes by leaning on declarative semantics, governed data products and robust checkpoint/versioning patterns in the lakehouse. Learn practical patterns to address bad input data, schema drift and logic errors — plus the observability and guardrails that stabilize streaming workloads as they evolve. Walk away with actionable designs, checklists and success criteria for resilient, cost‑efficient streaming at enterprise scale in regulated environments. Outcomes include faster recovery, lower operational effort and cost, improved reliability and correctness and sustained performance as concurrency grows.