New model drop? Simon Willison just shared notes on DeepSeek v4 Pro 0813. Here’s a fast, vendor‑neutral checklist to sanity‑check any fresh LLM in 30 minutes. Source: his post.
What to test in 30 minutes
- Real tasks over toys: 3 quick reps: fact‑based Q&A with citations, bug‑fix a small code snippet, and a spreadsheet formula or SQL transform.
- Instruction following: Give a multi‑step request with a strict format. Check if it follows order, style, and output schema.
- Grounding & citations: Ask for quoted evidence plus URLs. Verify links resolve and match the claim.
- Tools & JSON: If the model supports function calls, measure valid‑JSON rate, argument accuracy, and retries needed.
- Latency & cost: Log first‑token latency, tokens/sec, and estimated $/task. Keep temperature fixed.
- Safety balance: Probe with benign edge cases (medical, legal, finance summaries). Watch for over‑refusal or unsafe guidance.
- Non‑English & long context: Try a short bilingual task and one 20–50k‑token retrieval (if supported).
Simple, reproducible prompts
- Format lock: “Return only this JSON: {"answer": string, "sources": string[], "confidence": 0–1}. No extra text.”
- Source‑grounding check: “Answer in 120 words with two verbatim quotes and working URLs. If unsure, say ‘insufficient evidence.’”
- Tool‑call echo: “When using tools, always repeat the final arguments you intend to send as valid JSON.”
- Hallucination trap: Ask about a non‑existent paper/plugin. Expect a refusal or uncertainty, not made‑up details.
Quickbench harness (5 steps)
- Create a tiny CSV of 10 tasks (type, prompt, expected_format, evaluation_notes).
- Fix params: temperature=0.2, top_p=1.0, max_tokens set for task size.
- Log per run: first_token_ms, tokens_per_sec, output_bytes, valid_json (true/false).
- Auto‑check: JSON validity, presence of URLs, quote markers, and forbidden phrases.
- Save raw outputs with a timestamp for diffing.
Compare against baselines
- Use your production model as Control A.
- Add a strong open baseline (e.g., a modern Llama/Mixtral build) as Control B.
- Run the same CSV once per model; avoid prompt drift.
- Pick a winner per task, then tally. Note ties and reasons.
Red flags to watch
- Inconsistent function‑calling or schema breaks under pressure.
- Confident claims with dead or irrelevant links.
- High variance run‑to‑run at the same settings.
- Over‑refusal on harmless topics or easy jailbreaks on sensitive ones.
- Unclear or shifting rate limits that throttle throughput.
Trust but verify: resources
- Simon Willison’s notes on DeepSeek v4 Pro 0813: read here.
- Benchmarking basics and pitfalls: Stanford HELM overview: crfm.stanford.edu/helm.
Takeaway
New models deserve a fair shot, not blind trust. Run this 30‑minute, CSV‑based quickbench and only promote a model if it beats your controls on tasks you actually ship.
Get weekly, no‑fluff AI tactics in your inbox. Subscribe to The AI Nuggets.
