When things break, a tight, blameless incident report turns failure into compound learning. Here’s a copy-paste template—optimized for AI systems—you can ship fast.
For a real-world example of transparent reporting, see Simon Willison’s public incident write-up: Incident report.
A lightweight template you can copy
- Summary (2–3 sentences): What happened, when, and current status.
- Impact: Who/what was affected (users, data, revenue, SLAs), severity, duration.
- Timeline (UTC): First signal → detection → user impact → mitigation → recovery.
- Root cause: Technical cause(s) and contributing factors. Keep it blameless and factual.
- Detection: How it was detected (monitoring, user report, automated test, eval) and detection gap.
- Response: Actions taken, by whom, and why; include comms to users/stakeholders.
- Remediation: Immediate fixes, rollback, hot patches, and risk trade-offs.
- Prevention: Long-term changes, owners, deadlines, success metrics.
- Evidence & links: PRs, dashboards, runbooks, logs, evals, prompts, config versions.
- Open questions: Unknowns and a plan to close them.
AI-specific details to capture
- Model + version: Base model, finetune snapshot, or toolformer config at time of incident.
- Prompting: System and user prompts involved; retrieval context windows; truncation rules.
- Guardrails: Moderation settings, safety filters, jailbreak defenses, and their thresholds.
- Data & privacy: Any PII/PHI exposure, training/eval datasets touched, caching layers.
- Tools & connectors: Which actions/tools were enabled; rate limits; timeout and retry policy.
- Evals: Offline/online eval scores before/after; regression signals ignored or missing.
- Drift & rollout: Canary/AB gates, shadow traffic, model drift indicators, feature flags.
- Vendors: Third-party outages, API changes, pricing/quotas that influenced behavior.
Rapid fact-gathering prompts (15 minutes)
- What was the first user-visible symptom? Link a screenshot or log line.
- What changed in the last 24–72 hours? Code, config, model, data, traffic, vendors.
- What didn’t change but should have caught this? Alert, eval, test, or playbook.
- What was the fastest safe mitigation and why? Note trade-offs.
- What evidence rules out our top 3 alternative hypotheses?
Publishing checklist (ships in under an hour)
- Redact secrets and user PII; keep technical depth.
- Use UTC timestamps; include duration math.
- Embed or link artifacts: PRs, dashboards, runbooks, incident channel transcript.
- Assign owners and due dates for prevention items.
- Schedule a 30-minute blameless retro; capture 3 concrete follow-ups.
Why “blameless” works
Blameless postmortems increase signal and reduce fear-driven omissions. The goal is learning, not punishment—so you get better data and faster fixes.
Google’s SRE guidance popularized this approach; see their postmortem culture overview: Postmortem Culture: Learning from Failure.
Standards and references
- NIST AI Risk Management Framework: NIST AI RMF — structure your risk controls and incident learning loop.
- OWASP Top 10 for LLM Applications: OWASP LLM Top 10 — common failure modes to consider in your reports.
Takeaway
Use a short, repeatable template; capture AI-specific context; publish fast. Blameless documentation compounds learning and trust—with your team and users.
Get weekly, practical AI nuggets in your inbox: Subscribe to our newsletter.
