Simon Willison’s June AI newsletter is packed with pragmatic engineering notes. Here are three actionable lessons teams can apply this week—plus links to dive deeper (read the newsletter).
Why it matters
Models change weekly, but good engineering endures. These practices help you ship reliable AI features despite churn in models, tooling, and vendor APIs.
1) Treat structured output like an API contract
- Define a JSON schema and validate every response; use targeted repair prompts on failure.
- Prefer tool/function calls for deterministic fields; keep free text for rationale only.
- Set token caps and guardrails; stream and parse incrementally when possible.
- Log schema errors with examples and add unit tests that replay failing prompts.
Good starting point: OpenAI’s guide to structured outputs and function-style tool calls.
2) Retrieval beats long context: build a lean RAG
- Avoid dumping whole docs into prompts; retrieve the smallest helpful chunks.
- Chunk by semantics (headings/sections) and store source IDs for citations.
- Rerank top-k passages with a small cross-encoder before final generation.
- Evaluate retrieval separately from generation with a small, curated question set.
See Cohere’s concise RAG guide for practical patterns and trade-offs.
3) Evals that mirror reality
- Create tiny “canary” evals for your core tasks and run them on every change.
- Mix human-written edge cases with real user data (with privacy controls).
- Track latency, cost, and failure modes—not just correctness scores.
- Expect drift; re-run evals when models, prompts, or context sources change.
NIST’s AI Risk Management Framework offers a solid backbone for evaluation thinking (NIST AI RMF Playbook).
Quick-start checklist
- Add JSON schema validation and automatic repair to your next release.
- Swap “long prompt dumps” for a retrieval + rerank pipeline with citations.
- Stand up a 20–50 example eval suite and run it in CI on every change.
Takeaway
The winning teams treat AI features like backends: clear contracts, separation of concerns, and continuous tests. Do that, and model churn stops being scary.
Enjoy bite-sized AI insights like this? Subscribe to The AI Nuggets newsletter for weekly, practical updates.
