Simon Willison recently highlighted a post referencing Boris Cherny. It’s a timely nudge: shipping AI features that don’t flake is an engineering discipline, not a vibe. Here are seven practical moves you can apply today.
The playbook
- Narrow the job-to-be-done. Define one crisp user story and a success sentence (what a human would accept). Everything else is out of scope until v2.
- Write a system prompt as a contract. State role, allowed tools, output format, and refusal rules. Version it like code (v0.1, v0.2) and keep changes in git.
- Build a tiny eval suite. Start with 20-50 “golden” cases and score exactness, structure, and safety. Track a single pass rate you can show on a dashboard.
- Start small, scale on evidence. Begin with a smaller, cheaper model and promote to larger models only if evals prove a clear win per dollar and latency budget.
- Make outputs machine-checkable. Use JSON schemas, tool-calling, and few-shot examples. Keep temperature low for deterministic paths; raise only where diversity helps.
- Guardrails and citations by default. Filter unsafe inputs/outputs and prefer RAG that cites sources. If a claim can’t be sourced, define a polite refusal.
- Ship observability. Log prompts, versions, latencies, tokens, and eval outcomes with trace IDs. Add a lightweight human review queue for flagged cases and retrain weekly.
One-hour starter checklist
- Describe the single success criterion in one sentence.
- Draft a v0.1 system prompt with role, output schema, and refusal rule.
- Collect 25 real user cases; label 10 as hard negatives.
- Run baseline on a small model; record pass rate and latency.
- Add JSON schema validation and a safety filter; re-run evals.
Why this works
LLMs are nondeterministic dependencies. Specs, tests, and telemetry turn that uncertainty into manageable risk. Treat prompts like code and results improve fast.
Sources and further reading
- Simon Willison’s link to Boris Cherny: context and discussion
- Microsoft’s prompt engineering best practices: official guidance
- OpenAI Evals (examples of automated evals): GitHub repo
Takeaway
Constrain the problem, formalize the prompt, test on goldens, and instrument everything. Reliability is an engineering choice.
Get weekly, practical AI playbooks in your inbox—subscribe to our newsletter: theainuggets.com/newsletter.
