Streaming
Designing pipelines for the day you need to replay
A design exercise in recoverability: separate ingestion, durable history and serving so corrections have a clear path.
Start with the recovery question
Before selecting a streaming engine, ask what happens when a producer sends incorrect records for six hours. Define the retention window, the authoritative input and who can approve a replay. Throughput alone does not answer these questions.
Make writes repeatable
Give each logical event a stable identifier and make the destination handle repeated delivery explicitly. A replay should not silently double business totals. Keep event time separate from ingestion time so delayed events remain explainable.
Isolate the recovery path
Run backfills against a separate output or versioned dataset where possible. Compare record counts, key uniqueness and business aggregates before switching consumers. Keep the previous output available until validation finishes.
What to measure
Track lag, late arrivals, rejected records and reconciliation differences. A pipeline can be running while producing the wrong result. This is an illustrative design note, not a report of an employer system.