Microservices

Chapter 07 of 27 · Part 4: Transactional messaging and domain events

Domain events: design and schema evolution

How to shape the events services publish, and how to change them without breaking consumers.

key idea / coordinatorservice / good pathin progress / watch outproblem / failureneutral

Events sized for their consumers

How much should an event carry? Three common answers, shown with an order whose street number changed.

Three event sizesTown crierOrderStreetNumberChangedSmall and clear intentConsumer must call backGoldilocksOrderAddressChangedFull address includedLess exact intentDocumentOrderDocumentWhole entity, one eventIntent is lostTown crier is also called event notification. A document keeps the latest state per id.

A town crier event forces the inventory service to ask the order service for the rest of the address. That is a synchronous dependency, and it may even read stale data. Two ways out:

OptionWhat happensCost
Bee events (event-carried state transfer)Consumers keep their own copy of the data from every eventStorage, and copies spread. A new service can rebuild by replaying the stream
Goldilocks eventProducer adds the extra fields the consumers needProducer must know its consumers; do not add data from another domain
Documents on a compacted topicLatest document per key stays on the logConsumers lose the meaning of the change
Design with consumers in the room. If you cannot change an existing event (a legacy or external source), fall back to bee events first and town crier calls only as a quick fix. Very large events can also hint that a service has too much responsibility.

From Practical Event-Driven Microservices Architecture (ch. 3 and ch. 8).

How big should an event be?

Event size is a design choice with real trade-offs. The section above introduces it; here are the four named patterns. From Practical Event-Driven Microservices Architecture, ch. 8.

Town crier: small event, then askThe consumer receives a notification and calls the owner to fetch the rest.Order svcowns the addressBrokerOrderAddressChangedInventory svcneeds full addressDashed: a synchronous fetch. Simple to build, but inventory now fails when order is down.
Bee: carry the state with the eventConsumers keep a local copy of the fields they need and update it with each event.Order svcpublishes changesBrokerpartial eventsInventory svclocal order viewNo remote call, so no coupling at runtime. The copy holds only what inventory needs.
PatternIdeaTrade-off
Town crierSmall notification, consumer fetches the rest.Easy and quick, but a synchronous dependency returns.
BeeConsumers store the event data locally.More aligned with events; each service keeps a view.
GoldilocksEvent carries exactly what the consumer needs.Best fit when you control the schema.
DenormalizedA process manager merges streams into one larger event.Consumer sees one stream instead of several.
Denormalizing with a process managerA process manager merges order, stock and product events into one shipping event.Order eventsStock eventsProduct eventsProcess managerjoins by order idShipping eventone large streamThe shipping service no longer juggles concurrency across three sources.
Rule of thumb: if you own the event, aim for Goldilocks. If you cannot change it (a legacy or external source), prefer bee over town crier to avoid direct dependencies.

Event metadata and translation

Every event has a payload about the business fact. Information that every event shares, such as ids and times, belongs in headers or an envelope, like HTTP headers beside a body.

An event with metadata and payloadOrderCreatedMetadata (headers)Payload (the fact)event idcorrelation idsourceversiontimestampuser idOrderId: 3621ProductId: 2184Quantity: 2UserId: 164

Headers are optional and flexible. An envelope is a wrapper with a fixed shape, so it can force fields to be present, but changing it hurts every event. CloudEvents is a spec that standardizes this kind of data.

Between contexts, an anti-corruption layer translates events so one context's concepts do not leak into another. It keeps only what the inside needs.

Anti-corruption layerOutside eventtheir wordingTranslatoradapts and filtersInside eventour languageThe layer keeps domain logic and information inside the boundary.

From Practical Event-Driven Microservices Architecture (ch. 3 and ch. 8).

Headers, envelopes and contextual information

Events need metadata so consumers can trace, order and route them without opening the payload. You can carry it in broker headers or wrap the payload in an envelope. From Practical Event-Driven Microservices Architecture, ch. 8.

One event, two partsThe envelope or headers hold metadata, the body holds the business data.EventHeaders or envelopeid, correlation idsource, versiontimestamp, priority, user idBodyOrderId: 42Street: Main StNumber: 7Headers can be read without deserializing the body, so routing and filtering are cheap.
MetadataWhat it is for
Event idUnique per event. Lets a consumer detect duplicates (lesson 06).
Correlation idTies together every event caused by one request, for tracing across services.
Source and timestampWho published it and when, which helps debugging and ordering checks.
VersionTells consumers which schema the body follows.
Priority and user idPriority routes urgent events to a faster queue. User id supports auditing and erasure.
Standard first: CloudEvents defines a common set of such attributes, so tools from different vendors understand your events. Prefer it over inventing your own names.

Evolving schemas without breaking consumers

Events live in the broker for a long time, so every schema change must respect old consumers and old events. Lesson 06 shows the basics; this is the compatibility vocabulary. From Practical Event-Driven Microservices Architecture, ch. 8.

CompatibilityMeaningUpgrade first
BackwardNew schema can read data written with the old one.Consumers
ForwardOld schema can read data written with the new one.Producers
FullBoth directions work.Any order
NoneNeither works. Needs a coordinated cutover.Both together
Two-step breaking changePublish both versions side by side, migrate consumers, then retire the old one.v1 onlyall consumers on v1v1 and v2producer writes bothMigrateconsumers movev2 onlydrop v1Never remove a field the consumers still read. Add the new one first, then remove the old one later.
Stream versioning with an upscalerA converter service republishes old events as the new version so consumers only handle one.Topic v1Upscalerv1 to v2Topic v2Topic v2Downscalerv2 to v1Topic v1Top: new consumers read v2 only. Bottom: old consumers keep reading v1 during the migration.
Prefer evolution: a schema registry that enforces compatibility on publish catches most breaks before they reach production.

Choosing and evolving the event schema

Events live a long time, so their schema is as hard to change as an API. Four compatibility types describe a change:

TypeMeaningExample change
BackwardNew schema reads old eventsRemove UserName
ForwardOld schema reads new eventsAdd a country field
FullBoth directionsAdd description with a default value
NoneNeither can read the otherChange UserId to a UUID

Full compatibility needs default values, which Avro and Protobuf support and plain JSON does not. A schema registry can enforce these rules.

An incompatible change can be split into two safe ones. To turn an address string into a structured object:

Splitting an incompatible changeAdd new fieldforward compatibleBoth existconsumers move overRemove old fieldbackward compatibleSet a deadline for the old field so consumers have a fixed window to move.

For a deep redesign, publish each schema version on its own stream. Producers write v1 and v2 side by side, consumers move when ready, and old events are copied into the new stream. The cost is making sure both streams get the same events.

Stream per schema version with a downscalerOrder servicepublishes v2orders v2 streamNew consumers (v2)Downscalernew to old shapeorders v1 streamOld consumers (v1)

A downscaler is a separate component that turns new events back into the old shape. The domain service stays clean, and you switch the downscaler off once every consumer has moved. An upscaler does the reverse.

From Practical Event-Driven Microservices Architecture (ch. 8, Schema Evolution).

Recap

Quick check

The inventory service gets OrderStreetNumberChanged but needs the full address. Name two ways to avoid a callback to the order service.
Show answer

Have the order service publish a richer OrderAddressChanged event with the whole address, or let the inventory service keep its own copy of the address built from every order event (a bee event).

The inventory service must fetch the full address on every OrderAddressChanged event. What is the risk, and what are the options?
Show answer

It now depends synchronously on the order service and fails with it. Options: enrich the event (Goldilocks), keep a local copy (bee), or merge streams with a process manager.

Can you remove a field from an event that old consumers still read?
Show answer

Not directly. Add the replacement first, publish both versions, move consumers over, then drop the old field.

You must change an event field from a string address to a structured object without a big-bang release. What do you do?
Show answer

Add the new structured field next to the old one (a forward compatible change), let consumers migrate, then remove the old field (a backward compatible change) after an agreed deadline.