How incident response works
Incident management is the agreed process for restoring service when something breaks. Someone detects the problem and declares an incident with a severity. An incident commander coordinates while responders mitigate and a communications lead keeps users informed. Afterward, the team writes a blameless review and tracks the follow-up work to completion.
During the incident, restoring service comes first. Rolling back a release or failing over to a healthy region often ends user impact sooner than finding the root cause, which can wait for the review.
Detection and severity levels
Incidents start from alerts, customer reports or an engineer noticing something odd. Declare early: downgrading a false alarm costs little, while organizing a response late wastes the first minutes of an outage.
Severity levels set how fast and how widely to respond. Define them in advance by user impact rather than technical cause, and change the level as you learn more. A three-level scale might look like this:
- SEV1: a core service is down, or data is at risk, for many users. Page on-call at once, assign a commander and post public updates.
- SEV2: a major feature is broken or badly degraded, or a subset of users is affected. Page on-call and update the status page.
- SEV3: minor impact with a workaround. Handle it during working hours.
Roles during an incident
Separate coordinating from fixing. When everyone debugs and nobody coordinates, work gets duplicated and users hear nothing.
In a small team one person may hold two roles, but say so in the incident channel. Hand over the commander role explicitly at shift changes. The usual roles are:
- Incident commander: owns the incident, sets priorities, assigns work, approves actions such as a rollback, and declares the end. The commander usually doesn't debug.
- Communications lead: posts updates to users, support and stakeholders on a schedule.
- Responders: investigate and mitigate, and tell the commander about findings and planned changes before acting.
- Scribe: records the timeline. On small incidents the commander or communications lead does this.
Communicating with users
Acknowledge the problem quickly, even before you know the cause. Say what users are seeing, what is affected, any workaround, and when the next update will come.
Post updates on a fixed schedule, such as every 30 minutes, even when nothing has changed. Avoid guessing at causes or promising fix times you can't control.
When service is restored, say so and explain what happened in plain language. For major incidents, follow up with a short public summary once the review is done.
Investigating (13:58 UTC)
Some customers can't complete checkout. Card payments fail with an error
after the order summary. Orders paid by bank transfer are not affected.
Next update by 14:30 UTC.Keeping a timeline
Record events as they happen, with UTC timestamps: the first alert, when the incident was declared, severity changes, each action and its result, mitigation and resolution. Memory after a stressful outage is unreliable, and chat history is hard to read later.
The timeline is the backbone of the review. It also gives you time to detect and time to restore service, two measures worth tracking across incidents.
Blameless post-incident reviews
A blameless review asks how the system and process allowed the failure, assuming people acted reasonably with what they knew at the time. When people expect blame, they hide mistakes, and the real causes stay hidden too.
The review pays off only if its action items get done. Track them in the same issue tracker as other work and check the open ones in regular team meetings.
Hold the review within a few days, while details are fresh. A useful write-up covers:
- A summary of what happened and the user impact, including duration and who was affected
- The timeline
- Contributing factors, which are usually several rather than one root cause
- What went well during the response and what made it harder
- Action items, each with an owner, a priority and a due date