Simon Willison recently highlighted practical lessons from Sean Lynch on building with LLMs. Here’s a compact, field-tested playbook you can apply this week.
TL;DR: The playbook
- Define one success metric per feature (e.g., task success rate, edit distance, or time-to-first-value).
- Start with a tight, labeled eval set; expand with adversarial and real-world edge cases over time.
- Set latency budgets early; use streaming, caching, and tool calls to hit them.
- Constrain generation (structured outputs, JSON schemas, function/tool use) to reduce chaos.
- Instrument everything: traces, prompts, responses, user edits, and costs tied to IDs and versions.
- Guardrails are table stakes: content filters, input/output validation, and prompt-injection defenses.
- Ship in slices with feature flags and A/B tests; promote prompt/model versions like code.
- Close the loop: use user feedback and failure traces to retrain prompts and refresh evals.
How to implement this week
- Define success: Write a one-line product KPI (e.g., “70% pass@1 on our golden set under 2.5s p95 latency”).
- Build a golden set: 30–100 representative tasks with correct outputs. Add 10 “spicy” edge cases.
- Wire evals: Run nightly evals against your golden set; alert on regressions in accuracy, latency, or cost.
- Add structure: Require JSON output with a schema; validate before showing results to users.
- Budget latency: Pre-compute and cache frequent context; stream partial answers to keep UX snappy.
- Logs that matter: Capture prompt, model, tools used, latency, token counts, and user edits per session.
Metrics that actually matter
- Task success rate (pass@1 or assisted success after one refinement)
- Human edit distance (how much users must change before accepting)
- p50/p95 latency by path (pure LLM vs. tool-augmented)
- Cost per successful task (not per token)
- Deflection rate and retention for support/agent flows
Guardrails and risk
Apply input/output validation, content safety filters, and prompt-injection defenses—especially when tools or external data are involved. The OWASP Top 10 for LLM Applications is a solid checklist.
- Sanitize and bound inputs; strip or neutralize system prompts in user content.
- Use allowlists for tool arguments; validate and rate-limit external calls.
- Ground with retrieval or functions; prefer citations and structured outputs.
Common traps to avoid
- Optimizing prompts without a fixed eval set—leads to performative gains.
- Chasing model size instead of UX: latency and clarity usually win.
- Skipping versioning: prompts and tool graphs need semantic versions like code.
- One-shot launches: ship thin slices, instrument, and iterate.
Resources
- Context and discussion via Simon Willison’s note: Sean Lynch on shipping with LLMs
- Security checklist: OWASP Top 10 for LLM Applications
- Prompting foundations: Anthropic Prompt Engineering
Takeaway
Reliable LLM features come from discipline, not luck: define success, evaluate relentlessly, constrain outputs, and learn from every user edit.
Get more bite-sized, practical AI playbooks in your inbox—subscribe to our newsletter: theainuggets.com/newsletter.
