Simon Willison recently highlighted lessons from Armin Ronacher on shipping LLM features without drama. Here’s a distilled, do-this-now playbook for teams building with AI.
Ship LLMs like distributed systems
Treat the model as an unreliable, probabilistic service. Design around timeouts, retries, fallbacks, and clear contracts.
Keep prompts, data, and tools deterministic wherever possible. Make rollbacks, canaries, and audits boring and fast.
Production checklist (copy, adapt, ship)
- Prompts as code: template, version, diff, and review them in Git. Store system prompts with your app, not inside the model.
- Automated evals: run small, fast tests in CI with golden prompts, expected outputs, and acceptance bands.
- Prefer tools and structured I/O: use function calling and schemas over free text to reduce hallucinations.
- Grounding by default: use retrieval or policy-checked data sources; log citations for traceability.
- Failure first: timeouts, retries with backoff, circuit breakers, and graceful fallbacks to non-LLM logic.
- Ship safely: canary new prompts/models behind feature flags and include a one-click kill switch.
- Costs under control: budget by route/user, cap tokens, and alert on spend spikes; cache aggressively.
- Deep observability: log inputs, outputs, tools used, model/version, latency, and token counts; watch for drift.
- Human in the loop for high-stakes: require confirmation, dual control, or sample review before irreversible actions.
- Safety and compliance: filter PII/secrets, defend against prompt injection with allowlisted tools, and document data flows.
Minimal architecture you can ship this week
- Prompt registry and templates versioned in Git, surfaced via a simple service.
- Orchestrator that handles tool calls, retries, timeouts, and tracing (OpenTelemetry).
- Retrieval layer (vector + policy filter) that logs sources and enforces access rules.
- Evals suite in CI and a production canary with fast rollback paths.
- Dashboards tracking latency, quality signals, cache hit rate, and cost per request.
Sources
Further reading: Simon Willison on Armin Ronacher’s lessons; and the NIST AI Risk Management Framework for aligning engineering with governance.
The takeaway
LLMs make great copilots, not oracles. Treat prompts as code, add evals and canaries, enforce guardrails, and watch cost and quality like a hawk.
Get more fast, practical AI playbooks in your inbox—subscribe to The AI Nuggets newsletter.
