On-call and escalation
On-call is a duty, not a character test. If someone finishes a shift feeling like they failed, the process failed, not the person.
The rotation
Every rotation has two people: primary and secondary. Shifts run a week and hand over on Tuesdays, so the handover happens mid-week during business hours rather than on a Friday night.
Severity
| Level | Criteria | Response |
|---|---|---|
| SEV1 | Full outage or data loss | Immediate page, war room |
| SEV2 | Degraded for a subset of users | 15-minute response |
| SEV3 | Defect with no user impact | Business hours |
Acknowledgement and escalation
An unacknowledged page moves to the secondary after three minutes. After ten, it reaches the engineering manager.
mizban oncall ack INC-2291
mizban oncall escalate INC-2291 --to platform-lead --reason "needs database failover"
mizban oncall timeline INC-2291 --since 30mThe war room
A SEV1 has three roles, and nobody holds two of them:
- Incident commander: decides, and does not touch a keyboard.
- Operator: the only person running commands.
- Communications lead: posts a status update every ten minutes.
The worst incidents we have worked were not the ones with a subtle technical cause. They were the ones where three engineers were running commands against the same cluster without knowing about each other.
Afterwards
The post-mortem is written within five working days and names no individuals. Every action item gets an owner and a date, otherwise it is a wish.