Security veteran Thomas Ptacek’s stance on AI risk—recently highlighted by Simon Willison—boils down to this: treat LLMs as untrusted, high-variance components. Here’s a concise playbook to reduce blast radius without stalling delivery.
Why this matters
- Jailbreaks and prompt injection aren’t fully “fixable.” Plan for failure, contain impact.
- Regulators and customers expect auditable controls, not vibes-based safety.
- Guardrails improve reliability and cost control, not just security.
The core idea
- Treat the model like an untrusted interpreter.
- Minimize capabilities by default; grant tools and data on a just-in-time basis.
- Continuously evaluate and monitor outputs, not just prompts.
Practical guardrails you can ship this week
- Capability scoping: Map every tool/function to explicit permissions and inputs. Deny by default.
- Output allowlisting: Constrain to JSON schemas or enumerations where possible. Reject and retry on schema violations.
- Context isolation: Strip or sign retrieval context; never pass raw, user-supplied text to tools.
- Prompt hygiene: Use deterministic system policies. Assume secrets in prompts will leak—don’t put them there.
- Rate limits and circuit breakers: Cap requests, tool calls, and spend per session. Trip and degrade gracefully.
- Red-team and test suites: Maintain a regression pack of jailbreaks, injections, and policy tests. Run pre-merge and nightly.
- Audit and replay: Log prompts, model versions, tool calls, and outputs with request IDs for forensics and tuning.
Metrics that catch trouble early
- Jailbreak rate: % of sessions triggering policy violations or tool misuse.
- Schema violation rate: % of outputs failing JSON/format checks.
- Refusal/misuse ratio: Over- vs under-refusals relative to policy.
- Tool error ratio: Tool exceptions per 100 calls; rising trends often signal injection.
- Data egress anomalies: Unusually long outputs or sensitive-term matches.
One-minute rollout plan
- Wrap your model with a schema validator and output firewall.
- Introduce a tool permission matrix (who/what/when) and deny-by-default.
- Add request-level budgets and circuit breakers; alert on trips.
- Stand up a minimal red-team pack (top 10 jailbreaks) and run in CI.
- Turn on structured logging and a basic dashboard for the metrics above.
Sources
- Simon Willison on Thomas Ptacek’s perspective: simonwillison.net
- OWASP Top 10 for LLM Applications: owasp.org
- NIST AI Risk Management Framework (AI RMF 1.0): nist.gov
Takeaway
Assume LLM failure, then constrain capability, validate outputs, and watch the right metrics. That’s how you turn “clever attacks” into containable incidents.
Get smarter on AI, weekly
Enjoy concise, practical AI insights like this? Subscribe to our free newsletter: theainuggets.com/newsletter.
