“If you don’t understand, you can’t participate.” Simon Willison’s reminder applies to every AI decision on product, policy, and procurement. Here’s a one-week plan to get conversant—fast—so you can contribute with confidence.
You don’t need to be an ML engineer. You do need shared vocabulary, hands-on reps, and a simple way to measure what works. Invest one hour a day for seven days.
The 7-day AI participation plan
- Day 1 — Vocabulary that unlocks meetings: Learn tokens, context window, temperature/top_p, system vs. user messages, function/tool calling, embeddings, RAG (retrieval-augmented generation), hallucination, and evals.
- Day 2 — Touch the models: Use any reputable LLM interface. Give it 5 of your real tasks (summarize, draft email, extract fields). Note where it shines, fails, and what instructions improved outcomes.
- Day 3 — Try mini-RAG: Load 10–20 internal docs. Prompt: “Answer strictly from these docs; cite the filename.” Observe if it cites correctly and when it makes things up.
- Day 4 — Build a tiny gold set: Write 25–30 representative questions with correct answers. Use this as your benchmark to compare prompts, retrieval settings, and models.
- Day 5 — Safety and risk basics: Draft a one-page risk register: data exposure (PII, secrets), harmful outputs, bias, IP leakage, change management. Add 5 jailbreak tests you’ll always run.
- Day 6 — Observability: Log prompt, model/version, settings, tokens, latency, pass/fail vs. your gold set. A simple sheet is fine—trends matter more than tools.
- Day 7 — Decide and document: Write an Architecture Decision Record (ADR): problem, options, evaluation results, risks, next experiment. Share a 10-minute demo with stakeholders.
Questions to ask vendors and teams
- What evals did you run, on whose data, and what does “good” mean for our use case?
- How is PII handled? Data retention defaults? Can we turn off training on our prompts?
- What model/version are we on, and how will you notify us about changes?
- Show red-team results and mitigations for jailbreaks and prompt injection.
- What’s the rollback plan if quality drops or costs spike?
Lightweight practices that compound
- Create a shared glossary: One page, living document. Cuts meeting friction instantly.
- Keep a “prompt lab” file: Save best prompts, failures, and why they worked or not.
- Standardize a weekly eval run: Re-test your gold set after any model or prompt change.
- Bias and harm checks: Add 5 scenario prompts that reflect your customers and policies.
- Cost awareness: Track tokens per task. Note cheap wins where smaller models suffice.
Why this works
Hands-on reps + a tiny evaluation set turns opinion into evidence. You’ll speak in outcomes, not hype—exactly what executives and regulators need.
Resources
- Simon Willison: If you don’t understand, you can’t participate — a clear call for hands-on learning.
- NIST AI Risk Management Framework: official guidance for aligning AI activities with risk-aware practices.
Takeaway
Understanding beats spectating. Give yourself seven hours this week to build vocabulary, touch the tech, and measure results—so you can participate with authority.
Get weekly, practical AI playbooks in your inbox. Subscribe to The AI Nuggets Newsletter.
