AI incidents happen—hallucinations, prompt injections, data leaks, or model drift can impact users fast. Treat each one as a learning loop with a clear, repeatable report.
NIST’s AI Risk Management Framework recommends structured processes to identify, manage, and communicate AI risks. An incident report is the backbone of that process.
Public write-ups (for example, this recent incident report from Simon Willison) show how transparency accelerates learning across the industry.
Sources: NIST AI RMF, OWASP Top 10 for LLM Applications, AI Incident Database, and Simon Willison’s post.
What counts as an AI incident?
- User harm or safety risk caused or exacerbated by model output.
- Security or privacy exposure (e.g., sensitive data leakage via RAG or tools).
- Material business impact (downtime, refunds, SLO breach, reputational hit).
- Model performance regressions or unexpected behavior in production.
A lightweight AI incident report template
Copy this into your runbook and keep it to one page where possible.
- Summary: One-paragraph description of what happened and current status.
- Impact: Users affected, duration, severity (S0–S3), business metrics hit.
- Timeline: First detection, key events, fix, verification.
- Detection: How it was found (monitor, alert, user report) and signal quality.
- Root cause: Technical and process causes; include model, data, and toolchain factors.
- Contributing factors: Gaps in tests, guardrails, permissions, or reviews.
- Data affected: Types, sensitivity, jurisdictions, retention, and access scope.
- Models and prompts: Model family/version, prompt or policy versions, tool calls.
- Safeguards in place: Evals, rate limits, content filters, jailbreak defenses.
- Customer comms: What was told to users and when (attach templates if used).
- Remediation: Hotfix, rollback, or configuration changes; verification steps.
- Preventive actions: Tests/evals to add, policy updates, automation, owners, due dates.
- Owners: Incident lead, comms lead, on-call, approver.
- Severity and status: Current state, next review date, sign-off checklist.
- Artifacts: Links to logs, dashboards, PRs, eval runs, red-team transcripts.
Run the process in 5 steps
- Triage: Assign severity (S0–S3), open a ticket, spin up an incident channel.
- Contain: Disable risky tools, limit scope, roll back prompts or models if needed.
- Investigate: Pull logs, reproduce safely, diff prompt/model/data versions.
- Communicate: Stakeholder updates on a schedule; user notice if required.
- Learn & share: Publish the report internally; sanitize and contribute lessons to the AI Incident Database when appropriate.
Metrics that matter
- MTTD and MTTR for AI-specific failures (not just infra).
- Escape rate: % of unsafe outputs caught post-filter.
- Prompt-injection incidence and exploit reproduction count.
- Drift score deltas between staging and prod evals.
- Near-miss count and fix-time for guardrail gaps.
Common failure modes to watch
- Prompt injection and data exfiltration via tools or RAG.
- Training or retrieval data poisoning from untrusted sources.
- Insecure plugins/tools and excessive model agency.
- Broken auth and overbroad permissions on model endpoints.
- Supply-chain risks from third-party models or datasets.
See the OWASP Top 10 for LLM Applications for concrete categories and mitigations.
Tooling checklist
- Structured, queryable logs for prompts, tool calls, and outputs (with privacy controls).
- Data lineage and dataset versioning for retrieval and fine-tuning.
- Prompt registry with versioning and approvals.
- Evals before and after hotfix; include jailbreak and safety suites.
- Canary releases and feature flags for prompts, tools, and models.
- Guardrails for PII/secrets and safety policies at the gateway.
- “/incident” form that pre-fills the report and routes to on-call.
- Runbook stored with your code and linked in on-call rotation.
Governance and sharing
Decide what is internal-only and what can be shared. When possible, submit a sanitized version to the AI Incident Database to help the ecosystem learn faster.
Takeaway
A one-page, repeatable incident report turns chaos into improvement. Measure MTTD/MTTR, track root causes, and bake fixes into prompts, tools, and policy.
Like this? Get weekly, no-fluff tips in our newsletter: theainuggets.com/newsletter.
