Nkosinathi Mbambo
Spine build · Engineer
CDC head-to-head: Debezium, Airbyte, OLake
Three change-data-capture tools, one source, one workload, one harness: a decision record built before the answer is known.
- Status
- Harness ready · not yet measured
- Lead demand
- Engineer · Solutionist posture
- Spine plane
- L1 Evidence · L7 Execution & Feedback
- Method stage
- Compare · Consolidate
- Availability
- Harness ready
· What it gives you
If you want to choose a change-data-capture tool on evidence rather than habit,
A reproducible head-to-head on your own source, with a decision recorded per leg of the architecture and every figure traceable to a run.
I The problem
Change data capture is where the lakebase's truth becomes everyone else's events. Tools are usually chosen on a vendor page or a colleague's habit, and the choice then shapes the broker, the lakehouse and the cost of every replay for years.
II Approach
- Point all three tools at the same Postgres lakebase, with logical replication on and synthetic data only.
- Seed the same rows, then apply the same seeded workload of inserts, updates and deletes, each change carrying a unique sequence number so arrival can be checked exactly.
- Measure snapshot time, change-to-arrival latency (p50 and p95), burst throughput, ordering per key, delete and tombstone handling, a column added mid-run, kill-and-restart recovery (duplicates and gaps) and resource footprint.
- Write every number to a results file from the harness only; a value nobody measured prints as "not yet measured", never as an estimate.
- Decide per leg, not overall: the event leg and the lakehouse leg ask different things of change capture.
III Structure
- SourcePostgres lakebase, logical replication, synthetic rows
- WorkloadSeeded inserts, updates, deletes, one schema change
- ToolsDebezium Server → NATS JetStream (default) · Debezium → Kafka · OLake → Iceberg · Airbyte → Kafka
- HarnessLatency, ordering, deletes, schema, restart, footprint
- RecordResults table rendered only from measured files
· Results
Not yet measured. This table is rendered from the harness's result files; no number here is written by hand.
| Measure | Debezium | OLake | Airbyte |
|---|
| Initial snapshot | not yet measured | not yet measured | not yet measured |
| Change-to-arrival latency, p50 / p95 | not yet measured | not yet measured | not yet measured |
| Burst throughput | not yet measured | not yet measured | not yet measured |
| Ordering violations per key | not yet measured | not yet measured | not yet measured |
| Deletes and tombstones | not yet measured | not yet measured | not yet measured |
| Column added mid-run | not yet measured | not yet measured | not yet measured |
| Kill and restart: duplicates, gaps | not yet measured | not yet measured | not yet measured |
| CPU and memory footprint | not yet measured | not yet measured | not yet measured |
· Design fit, from the documentation
What each tool is built to do, read from its own documentation. This is not a measurement, and it is where the measurement will either agree or embarrass it.
Debezium
Log-based streaming of row changes into Kafka, which is the shape the event leg already has. A delete arrives as a delete plus a tombstone. Delivery is exactly-once in normal running and at-least-once after a crash, which is why the projector deduplicates by event id. Costs: row changes rather than decision events. The reference stack now runs it as Debezium Server, a single process that writes into NATS JetStream with no Kafka Connect cluster to operate; the Kafka Connect form remains for the comparison.
OLake
Built for Postgres to Apache Iceberg, with full refresh plus change capture through pgoutput and an upsert mode that deduplicates. It fits the analytical leg, history in the lakehouse, rather than the event bus: it runs as sync-and-exit batches, so its latency is a batch property.
Airbyte
An ELT platform with a UI, the broadest connector catalogue and scheduled syncs. It is the heaviest to run, and its Kafka destination appends without deduplicating, so the consumer must dedupe (the projector already does). It needs care with replication-slot retention, or a sync fails until the slot is recreated.
· Position, provisional
Decided per leg, pending the run. Debezium for the event leg, where order, deletes and replay matter most. OLake for the lakehouse leg, where volume into Iceberg matters more than milliseconds. Airbyte where the source is one of the many systems the other two don't reach. If the measurements disagree, the measurements win and this paragraph changes, with the reason recorded.
The reference stack and harness → The event-sourced search →
IV Craft and next
Debezium Server · NATS JetStream · OLake · Airbyte · Kafka · Postgres · Docker Compose
The harness, connector configs and compose stack are in the reference stack, validated but not yet run. The table fills when the stack runs on the in-house workstation; until then the position below is provisional and says so.
← All work The architecture →
Want this working in your organisation?
Bring the decision that keeps coming back. I will tell you plainly whether this instrument fits, what it would take, and what it would not do.
Start a conversation →or book a time →
