Frontier-lab agents are powerful—and attack-surface rich. Inspired by Simon Willison’s analysis of a frontier lab agent intrusion, here’s the practical kill chain and what you can ship now to block it.
The agent intrusion kill chain (in brief)
- 1) Entry: The agent ingests a poisoned page/file or a hostile API result containing prompt-injection instructions.
- 2) Instruction override: Malicious content convinces the agent to ignore safety rules or expand scope (“exfiltrate project docs”).
- 3) Tool abuse: The agent invokes high-privilege tools (file system, connectors, code exec) not needed for the task.
- 4) Privilege escalation: Compromised tool chains calls other tools or services with broader credentials.
- 5) Persistence: The payload stores long-lived tasks, notes, or schedules in agent memory or job queues.
- 6) Discovery: The agent enumerates drives, vector DBs, Slack/Drive/Notion spaces, or internal APIs.
- 7) Exfiltration: Data leaves via “allowed” channels (webhooks, email, pasted chat text, or covert DNS/HTTP).
- 8) Lateral movement: Access tokens and connectors pivot to other projects, environments, or tenants.
- 9) Anti-forensics: Logs are pruned, prompts are overwritten, and traces are routed through “normal” outputs.
Practical defenses you can ship this week
- Default-deny tools: Ship an allowlist per task and per environment. Block “write”, “exec”, and “network” by default.
- High-friction actions: Require human approval for file writes, external posts, repo pushes, or sending emails.
- Scoped, short-lived creds: Issue per-tool, time-bound tokens; rotate automatically and bind to IP/hostname.
- Egress controls: HTTP allowlists, DNS sinkholes, and MIME/type guards. No arbitrary IPs or raw sockets.
- Prompt-injection hardening: Strip/neutralize untrusted HTML/Markdown, disable inline tool-call markup from the wild.
- Provenance checks: Prefer signed, trusted data sources; label untrusted inputs and reduce tool permissions accordingly.
- Capability firebreaks: Split read vs. write tools; isolate “read-only RAG” from “writer/exec” agents.
- Rate limits and budgets: Cap tool calls, output size, token usage, and task duration per conversation.
- Safeguarded retries: No blind auto-retries after denials; require user confirmation with reason codes.
- Kill switch: A single toggle to revoke all agent creds and pause scheduled runs.
Design patterns to reduce blast radius
- Conversation sandboxes: New workspace per session (temp storage, temp keys, temp vector indexes).
- Least-privileged functions: Narrow, purpose-built tool schemas; reject free-text function names/params.
- Safe-mode fallback: If inputs are untrusted, disable write/exec tools and route through read-only pathways.
- Output containment: Sanitize agent output before re-ingest; block SSRF/URL-fetch from its own responses.
- Memory hygiene: Do not auto-persist “notes/tasks.” Require explicit user intent for long-term memory.
- Canaries and honey docs: Seed detectable markers; alert on any access or egress.
- Environment split: Separate internet-facing RAG from internal data agents; never share tokens across them.
What to log (append-only)
- Prompts, model responses, and tool-call JSON (redact secrets).
- Tool identities, scopes, and short-lived credential IDs.
- External domains contacted, request methods, sizes, and hashes of files moved.
- User approvals with reason codes; denials with policy names.
- Anomaly flags (unusual tool mix, sudden egress, prompt-injection indicators).
Further reading: the OWASP Top 10 for LLM Applications maps common risks and mitigations. For governance, see NIST’s AI Risk Management Framework.
Takeaway
Treat agents like junior engineers with root on day one: minimize capabilities, scrutinize untrusted inputs, and log everything. Most damage comes from tool abuse, not the model.
Like this? Get one practical AI nugget in your inbox each week. Subscribe to the newsletter.
