Streaming pipeline with quarantine and replay
Adds schema rejection, immutable capture, checkpoints and explicitly coordinated replay. Original worked design example; adapt responsibilities, thresholds and assumptions to your system.
What this template contains
- Graph
- Directed
- Edges
- 11
- Source
- 26 lines
Make it your own.
digraph G {
graph [rankdir=LR, bgcolor="transparent", pad="0.3", nodesep="0.45", ranksep="0.8"];
node [shape=box, style="rounded,filled", fillcolor="#edf4fa", color="#47728d", fontname="Arial", fontsize=12, margin="0.14"];
edge [color="#64748b", fontname="Arial", fontsize=10];
source [label="Source events"];
schema [label="Schema registry"];
ingest [label="Ingestion validator"];
raw [label="Immutable raw log"];
bad [label="Quarantine"];
stream [label="Stream processor"];
state [label="Checkpoint state"];
serve [label="Serving table"];
replay [label="Replay coordinator"];
owner [label="Data owner"];
source -> ingest [label="event"];
schema -> ingest [label="contract"];
ingest -> raw [label="accepted"];
ingest -> bad [label="invalid"];
raw -> stream [label="ordered stream"];
stream -> state [label="checkpoint"];
stream -> serve [label="upsert"];
owner -> bad [label="review"];
owner -> replay [label="authorized replay"];
replay -> raw [label="read range"];
replay -> stream [label="reprocess with version"];
}