Short, scrappy AI notes can become high-signal experiments. Inspired by posts like Simon Willison’s quick note “Cat and Thariq”, here’s a fast workflow to turn curiosities into shareable, reproducible results.
Why tiny AI observations are worth capturing
- They surface edge cases you’ll forget in a day if you don’t log them.
- They compound into regression tests and better prompts.
- They create public artifacts others can reproduce, improve, or refute.
A 15-minute workflow to go from note to experiment
- Capture: Paste the exact prompt, input, and output that surprised you. Include screenshots or raw text.
- Pin context: Record model, version/date, temperature, top‑p, max tokens, and any system prompt.
- Minimize: Strip extra context until the behavior still reproduces in <5 lines.
- Freeze: Set temperature low (e.g., 0–0.2) and fix a seed if the API supports it.
- Automate: Wrap it in a tiny script/notebook so one command re‑runs the test.
- Compare: Try at least one alternative prompt or model to check if the effect holds.
- Share: Publish the inputs, settings, and script so others can validate or extend.
What to record for reproducibility
- Model ID and provider (exact name and date—models change silently).
- Decoding settings: temperature, top‑p, top‑k, max tokens, frequency/presence penalties.
- System prompt and any tools/functions enabled.
- Context length, truncation rules, and retrieval sources (with links or hashes).
- Seed or determinism flags, if available.
- Time and region of the run (APIs can behave differently across regions/updates).
- Environment: SDK version, library versions, OS/arch, GPU/CPU if local.
Validate and share responsibly
As you publish, sanity‑check claims and disclose limitations. For risk‑focused experiments, align with the OWASP Top 10 for LLMs and broader governance in the NIST AI Risk Management Framework. Clear scope reduces hype and makes your result useful.
Takeaway
Don’t let interesting AI behavior evaporate. Capture it, minimize it, fix the settings, and publish a one‑click repro. Your future self (and the community) will thank you.
Get more practical AI nuggets in your inbox—subscribe to our newsletter: theainuggets.com/newsletter
