Jeremy Howard has long championed practical, data-first AI. Inspired by his body of work—and recent notes captured by Simon Willison—here’s a distilled playbook you can apply today.
Source: Simon Willison’s write-up and the fast.ai approach.
1) Start with a simple, strong baseline
- Compare against a dumb-but-tough baseline (e.g., majority class, BM25, or a small fine-tuned model) before you chase gains.
- Write a tiny test harness so every change is measured on the same eval set.
2) Let data beat cleverness
- Spend more time on labels, consistency, and coverage than on exotic architectures.
- Fix label errors, add hard negatives, and expand edge cases to move the needle faster.
3) Prefer small, fast models first
- Optimize for latency, cost, and reliability. If a compact model hits the target, ship it.
- Use transfer learning, distillation, and quantization before scaling up compute.
4) Measure what matters (not just accuracy)
- Track real KPIs: calibration, latency p95, throughput, dollar cost per 1k requests, and failure modes.
- Build a frozen “golden” eval set with representative production samples and tricky edge cases.
5) Ship with guardrails
- Add input validation, output constraints, allow/deny lists, and safe fallbacks.
- Keep a human-in-the-loop for high-risk actions and log decisions for auditability.
6) Make it reproducible
- Pin seeds, library versions, and datasets. Save config, code, and training commands together.
- Use notebooks for exploration; promote to scripts for production runs and CI.
7) Close the loop in production
- Capture user feedback, ranker clicks, and errors. Periodically refresh data and retrain.
- A/B test variants, monitor drift, and roll back quickly when quality dips.
Quick checklist: Ship something useful this week
- Define a golden eval set (100–300 real samples, including edge cases).
- Stand up a baseline (small fine-tune or rules + retrieval) and measure it.
- Fix 20 label errors, add 20 hard negatives, re-run eval, and log the delta.
- Add two guardrails (input filter + safe fallback) before exposing to users.
- Track latency and cost from day one; set a target SLO.
Why this matters
Practical beats perfect. This approach gets you value fast, reduces risk, and builds a compounding advantage as you learn from real users.
Further reading: Simon Willison’s notes and the fast.ai course.
Like nuggets like this? Subscribe to our newsletter for weekly, no-fluff insights: theainuggets.com/newsletter.
