Data7 min read

Apache Kafka

Build real-time data pipelines and event-driven architectures with Kafka - high-throughput, fault-tolerant, durable message streaming at any scale.

Apache KafkaEvent StreamingPub/SubData PipelineDistributed Systems

skill: Apache Kafka category: Data tags: [Apache Kafka, Event Streaming, Pub/Sub, Data Pipeline, Distributed Systems] readingTime: 7

Kafka is justified when you need high-throughput, durable event streaming with replay capability. For simple async tasks between 2 services, Redis pub/sub or BullMQ is simpler. For > 100K events/second, complex event routing, or multi-consumer fan-out with replay, Kafka is the right tool.

Workflow

Understand Kafka's core modelKafka's model is different from traditional queues - topics are append-only logs, consumers read at their own pace, and messages are retained for replay.
Produce messagesProducers write records to Kafka topics. Message keys determine partitioning - same key → same partition → guaranteed ordering for that key.
Consume messages in a consumer groupConsumer groups distribute partition load across multiple consumers - scale consumption by adding consumers (up to the partition count).
Process streams with Kafka Streams or FaustKafka Streams (Java) and Faust (Python) enable stateful stream processing - aggregations, joins between topics, and windowed computations - without external state stores.
Configure for reliability and performanceDefault Kafka config is safe but slow. Tuning producers and brokers improves throughput without sacrificing durability.

Output Format

Kafka setup: producer with acks=all, consumer group with manual commit, Faust stream processor, and monitoring.

kafka/
├── producer.py     # Event producer with delivery callback
├── consumer.py     # Consumer group with manual commit
├── streams.py      # Faust stream processing
└── topics.yaml     # Topic configs (partitions, retention, replication)

Topics:
  user-events: 12 partitions, 7d retention, RF=3
  order-created: 6 partitions, 30d retention, RF=3

Examples

Input

Output

Edge Cases & Gotchas

Consumer lag monitoring - Consumer groups falling behind producers create unbounded lag and potential data loss after retention expires. Monitor consumer group lag with kafka-consumer-groups.sh or Kafka Exporter + Grafana.
Exactly-once semantics - At-least-once delivery (retry on failure) can duplicate messages. For exactly-once, use Kafka transactions: producer.init_transactions() + consumer with isolation.level=read_committed.
Topic partition count is permanent - You can add partitions but never remove them. Over-partition (2x expected consumers) at creation time. Under-partitioning limits consumer parallelism.
Large messages - Kafka's default max message size is 1MB. For larger payloads, store the payload in S3/blob storage and put only the reference in Kafka.

When to Break These Rules

Use Redis pub/sub or BullMQ for simple task queues between 2 services - Kafka's operational complexity isn't justified for < 1000 events/second.
Use AWS Kinesis or Google Pub/Sub instead of self-managed Kafka when the team doesn't want to manage Kafka cluster operations - both are fully managed with similar semantics.

Reference Files

https://kafka.apache.org/documentation/
https://docs.confluent.io/