DevOps6 min read

Load Balancing & Scaling

Scale web applications horizontally with load balancers, auto-scaling groups, health checks, and session affinity strategies using Nginx and cloud load balancers.

Load BalancingNginxHorizontal ScalingAuto ScalingHigh Availability

skill: Load Balancing & Scaling category: DevOps tags: [Load Balancing, Nginx, Horizontal Scaling, Auto Scaling, High Availability] readingTime: 6

Scale horizontally (more instances) before scaling vertically (bigger instance) - it's cheaper, more resilient, and removes single points of failure. The prerequisite for horizontal scaling is stateless application servers - session state must live in Redis or a database, not in-memory.

Workflow

Design stateless application serversStateful servers can't be load-balanced - each request must go to the server holding that user's session. Stateless servers let any instance handle any request.
Configure Nginx as a load balancerNginx is a high-performance HTTP server and reverse proxy - handles millions of connections with minimal memory as a load balancer.
Set up health checksLoad balancers must detect unhealthy instances and stop routing to them - without health checks, users hit failed servers.
Configure AWS Auto ScalingAuto Scaling adds instances when load increases and removes them when it drops - handling traffic spikes automatically without paying for idle capacity.
Implement zero-downtime deploysDeploying by stopping all instances and starting new ones causes downtime. Rolling deploys replace instances one at a time.

Output Format

Load balancing: stateless app servers, Nginx upstream config, health check endpoint, AWS Auto Scaling, and rolling deploys.

Architecture:
  DNS → ALB (health checks every 30s) → ASG (2-20 instances)
  App servers: stateless, sessions in Redis
  Health endpoint: /health → 200 OK or 503
  Deploy: rolling, maxUnavailable=0, SIGTERM drain
  Scale out: CPU > 70%, scale in: CPU < 40%

Examples

Input

Output

Edge Cases & Gotchas

Sticky sessions (ip_hash) - ip_hash in Nginx routes the same IP to the same server - useful for WebSockets but breaks failover. Prefer stateless architecture + Redis sessions to avoid this dependency.
Connection draining - Removing an instance from the load balancer immediately drops active connections. Enable connection draining (AWS: deregistration_delay = 60s) to let in-flight requests complete.
SSL termination - Terminate TLS at the load balancer (not at each app server) to offload crypto work and centralize cert management. Traffic from LB to app servers can be HTTP on a private VPC network.
Long-polling and WebSockets - Long-lived connections (WebSockets, SSE) don't distribute well across round-robin load balancers. Use sticky sessions or Socket.io Redis adapter for pub/sub fan-out.

When to Break These Rules

Use serverless (Lambda, Cloud Run, Vercel) instead of managing load balancers and auto-scaling groups - they handle scaling to zero and to millions automatically.
Use a service mesh (Istio, Linkerd) instead of Nginx for microservices with complex load balancing requirements (circuit breaking, mTLS, canary routing per-service).

Reference Files

https://nginx.org/en/docs/http/load_balancing.html
https://docs.aws.amazon.com/autoscaling/