A new note from Simon Willison titled “Bad Codex bug” is a timely reminder: AI code generation can fail in subtle, production-breaking ways. Don’t trust outputs without guardrails. Source: Simon Willison.
What this means for builders
- Silent regressions: model updates or provider changes can shift output format and quality without notice.
- Non-determinism: the same prompt may produce different code, breaking brittle parsers and pipelines.
- Schema drift: “almost valid” JSON or slightly malformed code can cascade into runtime failures.
9 guardrails to prevent “Bad Codex”‑class bugs
- Contract your outputs: enforce a JSON Schema or Pydantic model and hard-fail on validation errors. Retries are fine, silent fixes are not. See Structured Outputs.
- Constrained tool use: prefer function-calling/tool-use APIs so models emit arguments, not free-form code. Anthropic guide: Tool Use.
- Canary prompts: pin a small suite of critical prompts; run hourly against current and candidate models. Alert on accuracy, validity, and latency deltas.
- Differential testing: compare outputs across two providers or model versions; block promotion on statistically significant regressions.
- Strict parsers and adapters: centralize parsing, add structured retries (temperature↓ + system reminder), and final fallbacks to safe defaults.
- CI for prompts: snapshot prompts, fixtures, and expected outputs; run in CI the same way you test code. Keep prompt versions in source control.
- Runtime monitors: track validation failure rate, output length, null/empty rates, and exception classes. Trip a circuit breaker on spikes.
- Shadow deploys and 1‑click rollbacks: ship new models behind flags, mirrored traffic only, auto‑rollback on SLO breach.
- Secure execution: sandbox any generated code with resource limits and no egress; scan outputs for insecure patterns. See OWASP LLM Top 10.
Minimal implementation checklist
- Add a JSON Schema and validator to every model call that returns structured data.
- Create 10–20 canary prompts; run on cron across current vs. candidate models; alert on failures.
- Wire CI to run prompt fixtures and block merges on contract breaks.
- Log structured metrics (valid/invalid, retry count, tokens, latency) and set SLOs.
- Gate new models behind a feature flag with instant rollback.
If you must ship today
Use structured outputs or function-calling, validate strictly, and add a small canary suite. That alone catches most “it looked fine in dev” failures.
Sources
- Simon Willison: Bad Codex bug
- OpenAI: Structured Outputs
- Anthropic: Tool Use
- OWASP: Top 10 for LLM Applications
Takeaway
Treat model outputs like untrusted dependencies. Define contracts, measure continuously, and make rollbacks cheap. Reliability is a process, not a prompt.
Like this nugget? Get more practical AI tactics in your inbox: Subscribe to The AI Nuggets.
