Incident response runbook
Use this runbook for platform-impacting incidents rather than isolated support cases.1. Detect and confirm
Use health checks, error trends, provider metrics, support reports, and request/run failures to confirm the impact is broader than one account.2. Create the incident
Record:- title
- affected components
- initial impact
- detection time
- status
INVESTIGATING - current evidence
3. Scope the blast radius
Determine:- user populations affected
- organizations/regions/deployments affected
- model/provider scope
- web vs Developer Platform
- background jobs affected
- billing/auth/storage dependencies involved
4. Mitigate
Choose the smallest action that reduces user impact safely:- disable an unhealthy route
- use capability-compatible fallback
- pause a failing feature
- stop/restart affected workers according to operational procedures
- enable maintenance when necessary
- roll back a recent configuration or release when supported
5. Communicate
Use factual updates: what is affected, what operators are doing, and what changed since the previous update. Do not publish unverified internal theories as confirmed root cause.6. Monitor recovery
After mitigation, move toMONITORING only when service indicators show recovery. Watch error rate, latency, queues, provider health, and representative workflows.
7. Resolve
MarkRESOLVED only after recovery has been verified—not merely because a fix was deployed.
8. Follow-up
For significant incidents, capture:- confirmed root cause
- detection gap
- mitigation effectiveness
- rollback/recovery behavior
- customer impact window
- prevention/follow-up actions
Timeline discipline
Preserve the timeline as an operational record. Do not rewrite old entries to make early uncertainty disappear; add later confirmed findings instead.Incidents
Review incident states, components, and timeline-writing guidance.