Insights · Modernization

The event backbone: the core of a microservices architecture

Event-drivenMicroservices

Key takeaway

A replayable event log at the center decouples services and gives an audit trail. Respect its contract: idempotent consumers, outbox, sagas.

Ask why a microservices estate turned into a distributed monolith and the answer is usually the same: services were deployed separately but integrated synchronously, one point-to-point REST call at a time. The failure is arithmetic. N services integrating pairwise means O(N²) connections, each with its own retry policy, timeout, and outage blast radius. Jay Kreps — the engineer behind Kafka — called this out directly in “The Log”: a central log replaces the point-to-point integration mess with one abstraction everyone subscribes to.

What the backbone actually is

The event backbone is an append-only, ordered, durable log — replicated, partitioned, replayable. Kreps’ argument for why this humble structure should anchor distributed systems is worth internalizing: a log decouples ordering from any physical clock; feeding the same ordered inputs to the same deterministic processors yields the same state (the state-machine replication principle); and the log-to-table duality means state is just a changelog materialized — which is why a replayable log doubles as your audit trail and your recovery mechanism. Kafka’s design documentation formalizes the operational contract: topics divided into totally ordered partitions, each partition consumed by exactly one consumer in a subscribing group, ordering guaranteed per partition, never globally.

That contract — not the marketing version — is what the rest of this article designs against.

Events, commands, and the coupling you’re signing up for

Not everything that crosses a broker is the same kind of message, and Martin Fowler’s four-pattern taxonomy (event notification, event-carried state transfer, event sourcing, CQRS) is the cleanest map of the trade-off space. The two ends matter most in practice:

  • Event notification — “something changed; here’s the ID, come ask me.” Minimal payload, minimal coupling, but the logical flow disappears from the program text: the sequence of calls now lives across subscriptions you can’t see in any one codebase. Fowler’s warning is earned: watch for the event used as a passive-aggressive command.
  • Event-carried state transfer — the event carries the full changed state, so consumers keep a local copy and never call back. More duplication, dramatically more resilience and lower latency. This is how reporting and read models stop being availability hostages of the producing system.

Choosing per use case — and recording the choice — is backbone governance. Kleppmann’s IEEE Data Engineering Bulletin paper makes the deeper argument: build applications from a replicated log plus a few simple primitives (Kafka, Samza, the Unix philosophy), and the system’s dataflows become inspectable and recombinable instead of entangled.

The traps, and the patterns that survive them

An empirical mining of more than 8,000 Stack Overflow questions on event-driven microservices (Laigner et al., arXiv 2024) found developers naming exactly the failure modes practitioners warn about — publishes that “always return true” even when the broker is down, events “processed in random order”, at-least-once duplicates, schema changes that break consumers. Consistency and decoupling were the top-cited concerns. Every one of these has a known, pattern-shaped answer:

  • The dual-write problem → transactional outbox. A service that commits to its database and then publishes to the broker will, eventually, do one without the other. The transactional outbox writes the event into the same transaction as the state change, and a separate relay (polling publisher or log tailing) publishes it — accepting that the relay may deliver more than once, which makes consumer idempotency mandatory, not optional.
  • Cross-service workflows → sagas. With database-per-service, two-phase commit is off the table. A saga is a reliably coordinated sequence of local transactions, undone by compensating transactions — choreographed through domain events or orchestrated through commands. This is the modern restatement of Pat Helland’s CIDR 2007 argument (“Life beyond Distributed Transactions”): scope work to single entities, coordinate between them with retried, idempotent, task-based messaging. Gregor Hohpe’s coffee-shop analogy (IEEE Software, 2005) is still the best intuition pump: real businesses run on asynchronous, buffered workflows, not distributed locks.
  • Schema evolution → registry with compatibility rules. Producers and consumers deploy independently; schemas must therefore evolve compatibly. A schema registry holding a versioned history per subject, with BACKWARD compatibility as the default gate (new consumers read old events; producers can’t break them), turns “the new release broke the downstream” into a CI failure instead of a production incident.
  • “Where’s the truth?” → event sourcing where it earns its cost. Event sourcing makes the log the system of record for a service: replay yields state, audit falls out, temporal queries become possible — and querying requires CQRS-style read models. Adopt it per service, where audit or replay genuinely pays; don’t mandate it estate-wide.

Why the backbone should be the core, not a component

  • Independent deployability becomes real. Producers don’t need consumers to exist yet. New subscribers join the stream without touching a single producer — the property point-to-point integration promises and never delivers.
  • Audit and recovery are structural. For a revenue system, “what exactly did we tell whom, in what order” is not a nice-to-have. A replayable log answers it by existing.
  • Load is absorbed, not propagated. A downstream service that’s slow gets its events when it recovers, instead of holding upstream threads hostage.
  • Analytics rides the same stream. The backbone is the landing zone for reporting, fraud, and ML pipelines without another integration project.

On the E-Tax modernization we governed the backbone decision by decision rather than as one monolithic choice: consumer autoscaling, topic retention, connector strategy each got its own record, because each is a separate trade-off with its own reversal cost. That is where an ADR practice earns its keep.

When the backbone is overkill

Honesty keeps the architecture credible. If you have a handful of services, synchronous calls, and no replay or audit requirement, a broker adds operational surface (clusters, partitions, schema discipline, consumer lag) without buying proportionate value. The backbone earns its centrality when integration count, audit obligations, or fan-out make the O(N²) mesh and the synchronous dependency graph the expensive option. Watch for the inversion: once the majority of new integrations are asynchronous, the backbone stops being an optimization and becomes the architecture’s load-bearing wall — at which point availability, retention, and schema governance are platform commitments, not side projects.

Discuss your event backbone

Sources