Idempotent backfill orchestrator: chunked date-range backfills with a durable SQLite ledger, lease-based crash recovery proven by a SIGKILL test, collision leases against live loads, and an Airflow DAG factory. Overhead measured at 0.24-1.7ms per chunk (13.8% on a real 365-day load).
python airflow etl sqlite reliability orchestration data-engineering data-pipelines exactly-once idempotency backfill duckdb data-pipelines-configurations
-
Updated
Jul 29, 2026 - Python