Leadership7 min read

Incident Command & Response

Lead incident response effectively with clear roles, communication protocols, decision frameworks, and blameless post-mortems.

incident-responseon-callreliabilityleadership

Incident command is the practice of leading technical teams through production incidents with clear roles, calm decision-making, and effective communication. Borrowed from emergency services, it reduces chaos and speeds resolution when systems fail.

Every incident needs a single Incident Commander (IC). The IC doesn't fix the issue-they coordinate. Communicate early and often. Time to resolve matters more than appearing perfect. Run blameless post-mortems to build organizational learning, not to assign blame.

Workflow

Declare and scope the incidentUnclear scope delays response; too-broad scope wastes resources; too-narrow misses impact.
Assign clear incident rolesUncoordinated responders duplicate work, miss actions, and overwhelm engineers.
Run a structured response loopPanic-driven debugging is slow; structured investigation is systematic.
Communicate proactively with stakeholdersSilence during incidents erodes trust faster than the incident itself.
Run a blameless post-mortemBlame silences information; blameless post-mortems surface the real root causes and systemic fixes.

Output Format

Incident declared with severity, roles assigned in Slack, 30-min status updates, mitigation completed, blameless post-mortem published with action items.

Examples

P0 declaration: "@here P0 Incident declared: Checkout service returning 500s. IC: @alice, Tech Lead: @bob. War room: #inc-2026-08-01-checkout. Status page updated. First update in 30 min."

Status update template: "**[14:30 UTC]** Investigating elevated error rates on checkout (40% 500s). Deployment at 14:15 is under investigation. No rollback yet-assessing. Next update: 15:00 UTC."

Five Whys: 500s → DB query timeout → missing index → migration script didn't run → missing CI check → action: add index validation to migration CI gate

Edge Cases & Gotchas

IC who is also the only engineer who knows the broken system: delegate IC role to someone else so the expert can focus on diagnosis-never let IC and technical lead be the same person during active diagnosis
Multi-team incidents: establish a cross-team IC coordinator; each team has their own IC reporting to the coordinator
During on-call burnout: mandatory post-mortem question "What would prevent paging for this?" to drive elimination of toil, not just resolution

When to Break These Rules

Small teams (<5 engineers) with simple systems may need lighter-weight incident processes; the key elements (declare, communicate, document) still apply even if role separation is informal.

Reference Files

Google SRE Book – Chapter 14 (Managing Incidents): https://sre.google/sre-book/managing-incidents/
PagerDuty Incident Response Docs: https://response.pagerduty.com/
Blameless post-mortem guide: https://www.atlassian.com/incident-management/postmortem/blameless