DevOps5 min read

Chaos Engineering

Test system resilience with chaos engineering - fault injection, chaos experiments with LitmusChaos, network degradation, and building confidence in failure handling.

Chaos EngineeringResilienceChaos MonkeyFault InjectionSRE

skill: Chaos Engineering category: DevOps tags: [Chaos Engineering, Resilience, Chaos Monkey, Fault Injection, SRE] readingTime: 5

Chaos engineering proactively injects failures into production (or production-like staging) to verify that your system handles them gracefully. The goal is not to break things - it's to find out what breaks before customers do. Start small, have a clear abort condition, and run during business hours when you have the team to respond.

Workflow

Define the steady state and hypothesisBefore injecting failures, define what "normal" looks like. The hypothesis: under experiment X, the system maintains steady state Y. Without this, chaos is just random destruction.
Run pod failure experiments with LitmusChaosLitmusChaos is a Kubernetes-native chaos framework - experiments run as Kubernetes resources, scoped to specific pods, and have automatic abort conditions.
Inject network failuresNetwork partitions are common in distributed systems - pods can't reach the database or a downstream API. Test that your circuit breakers and retries work correctly.
Test CPU and memory pressureCPU-saturated pods have high response latency. Memory pressure triggers OOM kills. Verify your health checks and autoscaling respond correctly.
Automate chaos in CI/CD with GameDaysAd-hoc chaos experiments are forgotten. Scheduled GameDays and automated chaos in staging CI build resilience systematically.

Output Format

Chaos engineering: hypothesis + steady state metrics, LitmusChaos pod/network/CPU experiments, AWS FIS for cloud-level failures, automated CI chaos with SLI verification.

Hypothesis: "Under X failure, system maintains Y metric"
Steady state: measure 10min before experiment
Experiment: LitmusChaos ChaosEngine + PODS_AFFECTED_PERC=50
Abort condition: error_rate > 5%
Automated: measure baseline → run chaos → verify SLI in CI

Examples

Input

Output

Edge Cases & Gotchas

Chaos in production vs. staging - Netflix Chaos Monkey runs in production. Most teams should start in staging with production-like traffic. Run production chaos only after building team confidence through many staging experiments.
Cleanup after failed experiments - If the LitmusChaos controller crashes during an experiment, the chaos may continue. Always set TOTAL_CHAOS_DURATION and monitor for cleanup. Have a manual abort procedure.
Communicating chaos to stakeholders - Chaos experiments in staging look like real incidents to monitoring tools. Notify the on-call team and update the status page with "maintenance" before starting to avoid false incident declarations.
Game Day scope creep - Chaos experiments tend to reveal multiple issues. Stay focused on the hypothesis; log other findings as separate tickets. Fix the primary finding before running the next experiment.

When to Break These Rules

Skip chaos engineering until you have basic observability (metrics, logs, traces, alerting) - chaos without observability is just random destruction with no learning.
Use simpler failure testing (kill one pod, restart one service) before adopting LitmusChaos - the full chaos engineering framework adds complexity not justified for small systems.

Reference Files

https://principlesofchaos.org/
https://litmuschaos.io/docs/