DevOps5 min read

Incident Response

Manage production incidents effectively - incident roles, communication, triage, runbooks, postmortems, and on-call best practices.

Incident ResponseSRERunbookPostmortemOn-Call

skill: Incident Response category: DevOps tags: [Incident Response, SRE, Runbook, Postmortem, On-Call] readingTime: 5

Incidents will happen. The goal is to minimize MTTR (Mean Time to Resolution), not prevent all failures. A good incident culture has clear roles (who decides, who communicates, who investigates), psychological safety (no blame), and learning (postmortems that identify systemic issues, not individual mistakes).

Workflow

Declare and classify incidentsEarly declaration avoids the confusion of multiple people independently investigating the same problem. Classification determines response urgency and escalation path.
Run the incident: roles and communicationClear roles prevent chaos - one person makes decisions (IC), one investigates (Tech Lead), one communicates externally (Comms). Role separation reduces cognitive load under pressure.
Investigate with structured triageUnder pressure, engineers jump to conclusions. Structured triage (timeline, hypotheses, elimination) finds root cause faster than random debugging.
Write blameless postmortemsPostmortems capture organizational learning from incidents. Blameless means focusing on system failures (lack of monitoring, missing runbook) not individual mistakes.
Build runbooks for common incidentsRunbooks eliminate guesswork under pressure - a step-by-step guide for the on-call engineer who's never seen this specific failure mode before.

Output Format

Incident response: SEV1-4 classification, IC/Tech/Comms roles, timeline-based triage, blameless postmortem, and runbooks with diagnostic + mitigation steps.

Declare: severity + impact statement + roles assigned
Update: every 15-30 min in incident channel
Triage: timeline → scope → hypotheses → eliminate
Postmortem: timeline + root cause + action items (owner + due date)
Runbook: symptoms + diagnostic steps + mitigation + escalation

Examples

Input

Output

Edge Cases & Gotchas

Alert fatigue - Too many alerts train engineers to ignore them. Audit alerts quarterly - any alert that fires more than once per week without action should be tuned, silenced, or removed.
Communication to customers vs. internal - Internal incident channels discuss root cause and blame-adjacent topics. Customer-facing status pages communicate only impact and ETA - never speculate on cause publicly.
Cascading failures - Mitigation steps can cause additional failures (restarting all pods simultaneously causes downtime). Use rolling restart, traffic shifting, or circuit breakers to mitigate safely.
On-call burnout - More than 2-3 pages per week causes burnout. Track on-call load per engineer. If load is high, reduce it by fixing underlying issues from postmortem action items - not by adding engineers.

When to Break These Rules

Adapt the runbook to the team's size - a 5-person startup doesn't need formal IC/Comms roles. Use PagerDuty + Slack + a postmortem doc; skip the bureaucracy until team size demands it.
Skip formal postmortems for SEV3/4 incidents - write a brief incident ticket comment with root cause and action items. Reserve full postmortems for SEV1/2.

Reference Files

https://sre.google/sre-book/managing-incidents/
https://www.pagerduty.com/resources/learn/incident-response/