Backend7 min read

Event-Driven Architecture

Design event-driven systems - Apache Kafka, SQS, message patterns, event schema evolution, idempotent consumers, and the outbox pattern.

Event-DrivenApache KafkaMessage QueueCQRSEvent Sourcing

skill: Event-Driven Architecture category: Backend tags: [Event-Driven, Apache Kafka, Message Queue, CQRS, Event Sourcing] readingTime: 7

Events decouple services - a publisher doesn't know who consumes its events, enabling independent evolution. The challenge is the lack of a synchronous contract: schema evolution, ordering guarantees, and at-least-once delivery require explicit design. Every event consumer must be idempotent.

Workflow

Design event schemas with evolution in mindEvent consumers can't be updated atomically with producers - old consumers must handle new event versions, and new consumers must handle old events still in the queue.
Produce and consume Kafka eventsKafka is the standard for high-throughput event streaming - durable, ordered within partitions, replay-capable, and consumer-group based.
Use the outbox pattern for reliable publishingWithout the outbox pattern, saving to DB and publishing an event are two separate operations - one can fail while the other succeeds, causing inconsistency.
Handle competing consumers and orderingMultiple consumer instances process messages concurrently for throughput - but messages for the same entity (same order ID) must be processed in order.
Handle dead letter queues and poison messagesA message that always fails processing (corrupt data, bug) blocks the queue. Dead letter queues capture failed messages for investigation without blocking processing.

Output Format

Event-driven: CloudEvents schema with schemaVersion, Kafka idempotent producer + idempotent consumer, outbox pattern for atomicity, partition by entity ID for ordering, and DLQ with exponential retry.

Event: { specversion, id, type, source, time, data: { schemaVersion } }
Outbox: INSERT INTO outbox IN SAME TRANSACTION as business write
Kafka: key=entityId (ordering) + groupId (competing consumers)
Idempotency: redis.setnx(processed:${eventId}) before processing
DLQ: maxReceiveCount=3 + exponential backoff retry topics

Examples

Input

Output

Edge Cases & Gotchas

Event schema registry - For large event-driven systems, use Confluent Schema Registry to enforce Avro/Protobuf schema compatibility. Producers register schema, consumers validate - prevents breaking changes from reaching production.
Consumer group rebalancing - Adding or removing consumer instances triggers rebalancing - all consumers pause while partitions are reassigned. Use cooperative rebalancing (kafkajs incremental.assignor) to minimize pause duration.
Clock skew in event timestamps - Producer machines may have clock drift - don't rely on event timestamps for absolute ordering. Use Kafka's message offset for ordering within a partition.
Kafka log compaction - Log compaction keeps only the latest message per key - useful for representing current state (changelog). For event streams where all events matter, disable compaction and set appropriate retention.

When to Break These Rules

Use SQS instead of Kafka for simple task queues without ordering or replay requirements - Kafka's operational complexity is not worth it for straightforward background job processing.
Use synchronous APIs instead of events when the caller needs an immediate response and the operation completes in < 500ms - async events add latency and complexity for operations that could be synchronous.

Reference Files

https://kafka.apache.org/documentation/
https://cloudevents.io/