Insights · Modernization

Zero-downtime data migration: expand, backfill, contract

CDCETL

Key takeaway

Expand, backfill, verify, contract: the parallel-change pattern makes data migration reversible, with no maintenance window in the plan.

The research framing is blunt about why this discipline exists: prior schema-change tooling forced teams to “carefully schedule downtimes for schema changes, sacrificing availability” (Hu, Wang & Zhou, VLDB 2023) — and a maintenance window scheduled around a revenue system is a bet that the migration finishes on time, which is exactly the bet that fails. The counter-discipline has a canonical name. Danilo Sato’s pattern on martinfowler.com is “parallel change, also known as expand and contract” — expand the interface to support old and new, migrate all clients, then contract by removing the old. Applied to databases, with Fowler and Sadalage’s evolutionary database design underneath — every schema change is a version-controlled migration script that keeps the database “backwards compatible with all previous application releases that are live in production” — this is how you move data while the system serves traffic.

The five stages, in order, each reversible

  1. Expand. Add the new column, table, or store alongside the old. The application writes both. Nothing reads the new copy authoritatively yet — this stage carries zero user-visible risk and reverses by doing nothing.
  2. Backfill. Move existing data in batches: idempotent (safe to re-run from any checkpoint), rate-limited (throttled below the latency budget), resumable. Stripe’s engineering account of migrating hundreds of millions of subscriptions objects runs this as “a large, multi-threaded migration” executed offline from the request path — “all the changes we made were incremental.”
  3. Verify. Dual-read: serve from the old source, read the new source in shadow, compare, alert on mismatch. Stripe used GitHub’s Scientist library to “read from both tables and compare results in production, alerting engineers on mismatches” — consistency checked continuously in production, not asserted once in staging.
  4. Switch reads, then writes. Behind a feature flag, first reads (instant rollback if the numbers disagree), then writes. The application never has a moment where it cannot serve; every switch is a flag flip, not a cutover.
  5. Contract. After a full business-cycle of clean observation, remove the old column or store — deliberately, and last. This is where reversibility ends, so it is where patience pays; the removal is as reviewed and scripted as the migration was.

Online schema change: two tool philosophies

For the “add a column to a 500-million-row table” class of change, two open-source approaches dominate MySQL estates, and their difference is instructive:

  • Trigger-based — pt-online-schema-change. Percona’s tool “alters a table’s structure without blocking reads or writes” by creating an empty modified copy, copying “in small chunks,” using triggers on the original to catch concurrent changes, and finishing with “an atomic RENAME TABLE operation.”
  • Triggerless — gh-ost. GitHub’s tool is “a triggerless online schema migration solution for MySQL that uses the binary log stream to capture table changes,” with throttling that “truly ceases writes on master” and dynamic reconfiguration mid-migration — triggers being, in the project’s judgment, “the source of many limitations and risks” (gh-ost).

The general lesson either way: the migration runs at the pace of a background copy with live throttling, not at the pace of a DDL lock. And for moving data between stores, log-based change data capture is the backbone: Debezium, “an open source distributed platform for change data capture,” “records all row-level changes within each database table in a change event stream” — read once from the transaction log, delivered in order into Kafka topics, where the event backbone fans them out to every consumer.

Rollback is designed, not hoped for

Each stage has a defined reverse:

  • Expand reverses by deletion (nothing consumed it).
  • Backfill reverses by stopping (idempotency makes the restart safe).
  • Read-switch reverses by flag (old source never stopped receiving writes).
  • Write-switch reverses by flag plus a reverse-backfill — the new source’s writes must be replayed to the old before resuming it as primary.
  • Contract has no reverse; that is why it is gated on a full observation window and why it is the only stage that ever needs a change-freeze conversation.

Define the rollback before each stage starts and write it into the migration runbook. A migration that discovers its rollback mid-incident is performing an outage with extra steps.

The data half of the strangler fig

Zero-downtime migration is also the pattern that makes incremental system replacement possible. The strangler fig article describes the traffic façade; this is what happens behind it: per domain, backfill into the new store, keep it synchronized with CDC from the monolith, validate with dual-reads, switch the new system over, and only then retire the legacy tables — with the explicit warning that removing legacy objects ends cheap rollback. The data-layer lesson underneath both articles is the same: migrations fail at data because teams treat schema as a one-weekend event instead of a continuously-evolving, backwards-compatible surface. Academic work continues to push the frontier — treating schema changes as data modifications on snapshot-isolated storage makes online evolution “almost free” (VLDB 2023) — but the empirical base remains thin (schema-evolution studies are “only scarcely available in database research,” and famously hard to replicate), so production discipline, not published benchmarks, is still what keeps you up.

Rule of thumb

If the migration plan contains a maintenance window, it is not a zero-downtime migration — it is an outage with a press release. Expand, backfill, verify, switch, contract: five reversible stages beat one irreversible night.

Move data without moving downtime

Sources