Distributed Systems Fundamentals

Workflow

1. Know the eight fallacies of distributed computingThe most common distributed system bugs come from assuming the network works like local function calls. These assumptions are all false in distributed systems.
2. Implement idempotency keysAt-least-once delivery means your consumers receive messages multiple times. Every operation that changes state must be idempotent - applying it twice produces the same result as applying it once.
3. Use logical clocks for orderingPhysical clocks drift - you can't use timestamps to determine causality in distributed systems. Lamport timestamps and vector clocks establish causal ordering without synchronized clocks.
4. Handle split-brain with leader electionWhen the network partitions, two nodes may both believe they're the leader and accept conflicting writes. Raft consensus ensures only one leader exists per term.
5. Design for partial failures with bulkheadsA slow dependency (database, external API) blocks all threads/connections waiting for it - cascading failure. Bulkheads isolate resource pools so one failure doesn't consume all resources.

Output Format

Distributed systems: eight fallacies design implications, idempotency key caching in Redis, vector clock causality, Kubernetes leader election with Raft, and bulkhead isolation per dependency.

Design: every call needs timeout + retry + circuit breaker
Idempotency: redis.setex(idempotency-key) → cache response
Ordering: LamportClock.tick() / VectorClock.merge()
Leader: leaderelection.run(lock, { onStartedLeading, onStoppedLeading })
Bulkhead: maxConcurrency per downstream + BulkheadRejectedError on overflow

Examples

Edge Cases & Gotchas

• Network timeouts vs. application timeouts - TCP keepalive and socket timeouts are at the OS level. Application-level timeouts (axios timeout: 3000) are needed in addition - otherwise the socket hangs indefinitely if the server accepts the connection but never responds.
• Thundering herd after circuit breaker reset - When the circuit breaker moves to half-open and lets one request through, many callers may rush simultaneously after it succeeds. Implement gradual admission (1% → 10% → 100%) instead of immediate full open.
• Two generals problem - It's impossible to guarantee exactly-once delivery over an unreliable network - this is proven impossible (Two Generals problem). Design for at-least-once + idempotency instead of trying to achieve exactly-once at the network level.
• CRDT for conflict-free replication - CRDTs (Conflict-free Replicated Data Types) resolve conflicts automatically - counters, sets, and text (similar to Google Docs) that merge without coordination. Used in distributed databases, collaborative editors, and offline-first apps.

When to Break These Rules

• Use a managed distributed service (Kafka for messaging, etcd for consensus) instead of implementing distributed algorithms yourself - implementing Raft correctly is a multi-year project.
• Accept eventual consistency for non-critical data rather than implementing full distributed transactions - the operational complexity of two-phase commit across services rarely justifies the benefit.

Reference Files

• https://www.allthingsdistributed.com/
• https://raft.github.io/