Blog

What If Database Replication Were Designed Like a Vectorized Query Engine?

Why Supermetal treats CDC as an unbounded query over Arrow record batches, with parallel scans, native Parquet writing, and object-storage durability.

Database replication looks simple from a distance. Read a transaction log, copy the existing tables, and apply each change somewhere else.

The difficult part is everything between the source and the target. A replication engine must move an unbounded stream through systems that fail independently. It must preserve database types and transaction order, limit memory use, survive target outages, and resume without losing or duplicating data. During the initial snapshot, it must do all of this while reading billions of rows quickly enough to catch up with the live transaction log.

A recent profile of CedarDB begins with a first-principles question. What would a database look like if it were built for modern hardware rather than inherited assumptions?

For replication, the equivalent question is:

What if database replication were designed like a vectorized query engine?

Supermetal's answer is one in-process operator pipeline whose unit of execution is an Arrow record batch, with a durable payload buffer backed by object storage.

What is hard about replication?

An initial snapshot and continuous change data capture are two phases of the same operation.

The snapshot establishes a consistent source position and reads the existing state while the source continues to accept transactions. When the scan ends, log reading must resume at the exact position established before the scan. No change can fall into the handoff.

Continuous CDC has no end. The engine must know whether the target committed an in-flight batch before it records progress. It must also preserve transaction boundaries and schema changes across restarts.

This is why replication architecture is more than connector code. These architectural decisions determine both performance and correctness.

How CDC got its architecture

The row message became a natural unit for data integration. Sources produce records, connector interfaces expose records, and targets apply records. Debezium and Kafka Connect made this architecture canonical for open source CDC. Debezium decodes database transaction logs into change events. Kafka stores those events in durable topics, and sink connectors or stream processors apply them downstream.

The row-message pattern generalized beyond this stack. A process boundary isolates connectors from one another. An orchestrator manages their lifecycles. A broker or staging layer lets the source and target run independently.

These choices solved real product requirements. They supported many sources and targets, connectors written in different languages, distributed execution, and fault isolation between connectors.

They also created a long data path. A decoded database row can become a connector object, a serialized message, a broker record, a staging-table row, and finally an input to a bulk loader. Each boundary can introduce another representation, queue, network transfer, or type conversion.

The target rarely wants the intermediate representation. A database usually wants its native binary bulk-load protocol. A warehouse or lakehouse often wants typed columnar files such as Parquet. Before the target can load the data, the pipeline must collect rows, recover their types, build columns, and encode the final format.

Recent work from established data movement systems shows where this architecture is under pressure.

Their architectures differ, but their engineering work points to the same performance bottlenecks. How much CPU time goes into serialization and type conversion? How much data must sit in memory? How much work can run in parallel?

The next question is not only how to make each row cheaper to move.

Should the row remain the unit of execution?

What query engines learned

Analytical databases faced a similar question. Tuple-at-a-time execution paid interpretation and function-call costs for every value. Modern engines amortize that work across columnar batches and apply the same operation to many values at once.

DuckDB, Apache DataFusion, and ClickHouse move columnar batches through streaming operators. They divide scans into small units of work and schedule those units across CPU cores. Memory stays bounded by the working set instead of the complete input.

Supermetal is written in Rust and was built from the ground up as a vectorized streaming engine for replication, inspired by DuckDB and Apache DataFusion. Its unit of execution is an Apache Arrow record batch.

The source pipeline decodes records into typed Arrow columns. Subsequent pipeline stages exchange those columns as record batches within one process. The final stage produces the format the target loads.

Supermetal CDC ArchitectureA continuous ticker shows source database WAL transactions. Supermetal reads the WAL, batches transactions, and writes table-level Parquet files to object storage. Supermetal sends load and merge SQL to the target, which reads those files directly from object storage.Supermetal CDC ArchitectureWAL changes become table-level Parquet files loaded directly from object storageSource Database WALBEGINorders↻ UPDATECOMMITBEGINcustomers↻ UPDATECOMMITBEGINorders+ INSERTorder_items+ INSERTcustomers↻ UPDATECOMMITBEGINorders↻ UPDATECOMMITBEGINcustomers↻ UPDATECOMMITBEGINorders+ INSERTorder_items+ INSERTcustomers↻ UPDATECOMMITBEGINorders↻ UPDATECOMMITBEGINcustomers↻ UPDATECOMMITBEGINorders+ INSERTorder_items+ INSERTcustomers↻ UPDATECOMMITSupermetal CDCRead WALBatch transactionsWrite Parquet by tableTargetAPPLY LOAD BATCHordersMERGEorder_itemsMERGEcustomersMERGESQL load / mergeObject Storage Bufferpublic.orders/_sm_dt=2026-08-30/…parquetpublic.order_items/_sm_dt=2026-08-30/…parquetpublic.customers/_sm_dt=2026-08-30/…parquet
Supermetal reads the source WAL, writes table-level Parquet files, and instructs the target to load and merge those files from object storage.

A snapshot is a parallel table scan

Before a snapshot begins, Supermetal discovers the source tables and divides each table into scan ranges. Postgres scans use ctid page ranges, and Oracle scans use ROWID ranges. When a source does not expose a suitable physical row locator, Supermetal partitions the scan by primary key, using a clustered index when available.

Each range is an independent unit of work scheduled across the available scan capacity in the style of morsel-driven execution. The scheduler distributes work across tables and ranges, allowing small tables to make progress while large tables continue in parallel.

Each range streams from the source as Arrow record batches. Memory use depends on the batch size and the number of active scan slots, not the table size, so the engine does not need to hold a complete table in memory.

One columnar path

Adjacent stages exchange Arrow record batches directly in memory rather than serializing rows across process and network boundaries.

Arrow preserves database type information such as decimal precision and scale, integer signedness, and timestamp resolution. A target does not need to infer that information from JSON or recover it from strings.

For warehouse and lakehouse targets, the columnar layout carries through to Parquet encoding. Arrow's native Parquet writer applies each column's encoding to arrays of typed values. Supermetal can encode independent columns in parallel on a work-stealing pool while I/O proceeds independently. The writer receives column buffers ready for encoding, avoiding the work of unpacking intermediate row messages and rebuilding their columns.

For database targets, the final operator uses the target's native binary bulk-load protocol.

Supermetal does not make row transport incrementally cheaper. Its operators consume and produce columnar batches until the final target-specific encoding. For file-loading targets, it materializes data once as target-native Parquet.

Row-message input vs. columnar batch input
Row-message input
order_idskuquantity
1042A172
1043B047
1044A171
1045C224
Assemble column buffers
order_id1042104310441045
skuA17B04A17C22
quantity2714
Columnar batch input
Arrow record batch
order_id1042104310441045
skuA17B04A17C22
quantity2714
Native Parquet writer
Row-message input
Columnar batch input
The row-message path must transpose records into typed column buffers before encoding. An Arrow record batch already provides those buffers to the Parquet writer.

Parallel scans keep CPU cores hot. Columnar encoding streams through input batches while I/O proceeds independently. Together, these architectural choices reduce snapshot and backfill times to minutes.

Rust helps, but it is not the execution model. A row-at-a-time pipeline rewritten in Rust still pays per-record interpretation, allocation, and serialization costs.

Object storage is the durable exchange

Vectorized execution explains how data moves through Supermetal. Object storage makes the batch payloads durable without putting a broker on the data path.

This is part of a broader change in systems architecture. Neon separates Postgres compute from a storage layer that retains immutable history in object storage. SlateDB delegates durability to object storage and separates readers, writers, and compactors. turbopuffer keeps its durable state in object storage while RAM and NVMe provide faster cache layers. Cursor stores its Git write-ahead log in S3 and treats repositories on local disk as warm caches.

Each system adds storage and coordination machinery that determines which writes are durable and in what order. Database replication starts with an ordered, authoritative change log maintained by the source. Supermetal does not insert a second streaming log between the source and target.

Supermetal uses object storage as the durable payload buffer between the source and target. The payload format depends on the target. File-loading targets use target-native Parquet written to S3, Google Cloud Storage, or Azure Blob Storage. Database targets use buffered Arrow batches before native binary bulk load. The buffer can also use local NVMe or a filesystem when the deployment does not require cloud object storage.

During CDC, Supermetal batches transactions into Parquet or Arrow files per table. A single setting, flush_interval_ms, controls how often buffered changes are published, balancing latency, cost, and throughput. Shorter intervals favor freshness, while longer intervals amortize object-storage and target-load overhead across more transactions.

As the Parquet encoder flushes, Supermetal streams its output to object storage through multipart uploads rather than holding the full window of changes in memory. It keeps pointers to the files in its own buffer metadata, avoiding slow, costly metadata lookups against object storage.

If target loading stops, the buffer retains completed files until loading resumes. The source reader and target writer do not have to operate in lockstep.

This architecture also determines what customers operate.

Supermetal runs as one binary inside customer infrastructure, backed by object storage. There is no separate broker cluster or external orchestrator to deploy and coordinate.

A query engine that never stops

A snapshot has a finite input. Continuous CDC does not.

After the snapshot, Supermetal reads the transaction log from the consistent source position established before the scan. It decodes inserts, updates, and deletes into Arrow batches. Postgres logical replication and MySQL binlogs expose transaction boundaries directly. Oracle LogMiner can interleave changes from concurrent transactions, so Supermetal reconstructs those transactions and preserves their commit order before batching them for the target.

A target can use source transaction boundaries for atomic multi-table application when configured. Supermetal can also batch several source transactions into one target load without splitting a source transaction.

Source and target databases can restart or fail over independently, and network partitions can interrupt replication. Supermetal automatically reconnects and retries the interrupted work, so brief outages do not require operator intervention. It checkpoints progress after the target commits and uses source-derived row versions to make replay safe.

With schema migrations enabled, supported changes move through the same running pipeline. Arrow schema metadata carries the new shape, and the target writer applies the compatible target change.

Supermetal tests replication correctness the way query engines test query correctness. It adapts sqllogictest to apply transactions and schema changes to real source databases, replicate them, and compare source and target results after commits, crashes, and replay. A second fuzzing harness sends randomized values for every source type through the pipeline and fails on a change in value or precision.

We borrowed the execution model, not the entire query-processing problem. Replication does not need a SQL parser, cost-based join optimization, or arbitrary operator plans. Its physical path is narrower. It scans or decodes, buffers, encodes, and applies.

Supermetal handles transaction ordering, checkpointing, and recovery inside the replication engine.

A CDC engine is a query engine that never stops.

The execution model is the architecture

Twenty years ago, MonetDB/X100 showed how vectorized execution avoids tuple-at-a-time overhead. We built database replication on the same principle.