Writing an incident plan before you need one

Written By: BrainSoft In Security

Most teams write their incident plan during the incident. Someone notices the checkout is down, a call starts, and for the next hour people argue about who should be doing what while customers keep hitting a broken page. The plan gets written afterwards, usually as a list of apologies and a vague promise to do better.

An incident plan is a short document you write on a calm Tuesday, not a Friday night. It names severity levels, assigns roles, and lists the first five things anyone does when an alert fires. Below is how I build one for a project, usually in under two hours.

Start with severity levels, not with tools

The first argument in any incident is whether it is serious. Settle that in advance with a small table. Four levels is plenty. The point is that anyone on call can read a symptom and pick a level without asking a manager.

SEV1: customers cannot use the product at all, or data is being lost or exposed. Page everyone, wake people up.
SEV2: a major feature is broken for many users, or performance is degraded badly. Page the on-call engineer.
SEV3: a feature is broken for some users, or there is a workaround. Ticket it, handle in business hours.
SEV4: cosmetic or internal-only. Normal backlog.

Write one concrete example next to each level from your own system. "SEV1: login endpoint returns 5xx for more than 5% of requests for 3 minutes" is more useful than "critical outage".

Assign roles by name, then by backup

During an incident, one person should be fixing things and a different person should be coordinating. When the same person does both, the fix slows down and nobody talks to anyone. Name three roles and put real names and phone numbers against them.

Incident commander: runs the call, decides severity, decides when to escalate. Does not touch the keyboard.
Responder: the engineer doing the actual debugging and fixing.
Communicator: updates the status page, support, and stakeholders. Keeps a timestamped log.

If your team is three people, the same person can hold two roles, but write down which two and in what order they drop them. A plan that assumes ten people is useless to a team of four.

Write the first five minutes as a checklist

The worst moment to think is the first minute after a page. Give people a checklist that works before they understand the problem. Ours usually looks like this, and it lives in the repo next to the code.

## First 5 minutes
1. Acknowledge the page in PagerDuty within 5 min.
2. Open #incident channel, post: symptom, time, severity guess.
3. If SEV1 or SEV2, call the incident commander.
4. Check the status page and last deploy time.
5. If the last deploy is within 30 min, roll back first, debug after.

That last line has saved more incidents than any dashboard. Rolling back is reversible and fast. If we are not sure, we go back and investigate from a known-good state. Write that permission into the plan so nobody waits for approval.

Keep runbooks short and link them

A runbook is not a design document. It is the commands you run at 3am when the database is saturated. Keep each one to a page and test it occasionally.

# Check whether the API is healthy
curl -sf https://api.example.com/health | jq .status

# Recent deploys
kubectl rollout history deploy/api --namespace prod | tail -n 5

# Roll back the last deploy
kubectl rollout undo deploy/api --namespace prod

Link each runbook from the plan by symptom, not by service. "Checkout is slow" should point to the database runbook, the cache runbook, and the provider status page. If you are running this on infrastructure we manage, the our services team keeps these runbooks alongside your environments.

Write the postmortem template now too

The plan should end with what happens after. A blank template forces the right questions: what did customers see, when did we know, what did we do, what will we change. Set a rule that SEV1 and SEV2 get a written postmortem within five working days, and that the action items get owners and dates. Otherwise the document is just a story.

Store the plan where people will find it during an incident: pinned in the on-call channel, linked from the alert itself, and in the repo. Review it every quarter and after every SEV1. If a step was wrong, fix it that week. If you want a second pair of eyes on yours, get in touch.

Frequently asked questions

How long should an incident plan be?

Two to four pages. Severity levels, roles with names and backups, the first-five-minutes checklist, links to runbooks, and the postmortem rule. If it takes more than ten minutes to read during an incident, it is too long.

Who should own the incident plan?

The person who is on call most often, not a manager. They know which alerts are noisy and which steps are unrealistic. A tech lead reviews it, but the on-call engineer writes it and keeps it current.

What if we are a small team with no dedicated on-call?

Write a smaller plan. One severity table, one named responder with a backup, and a checklist that starts with checking the last deploy and rolling back. That covers most real incidents even without a formal rotation.

#Security