Want Claude to produce the same style tomorrow as today? A recent post by Simon Willison sparked a common question: how do you make a creative pattern feel “permanent” without turning your workflow brittle?
Why LLM outputs drift
- Provider changes: models get silently upgraded, safety policies tighten, tokenizers shift.
- Randomness: temperature, top-p, and sampling seeds vary across runs and providers.
- Context volatility: retrieval results, memory states, and tools change underneath you.
- Prompt creep: teams tweak instructions without version control or tests.
The permanence checklist (Claude + any LLM)
- Pin exact model versions, not “latest.” Record full IDs (e.g., provider + model + snapshot date) in code and docs.
- Fix generation settings: temperature, top_p, max_tokens. Use lower temperature for repeatability.
- Use seeds if available. Some providers support a sampling seed; if yours doesn’t, lean on structure and tests.
- Structure outputs. Ask for JSON with a schema or strict headings to constrain variance.
- Version prompts. Store system and user prompts in Git with change logs. Treat them like code.
- Freeze retrieval. Version your embeddings model, index timestamp, and data source commit SHA.
- Create golden tests. Save canonical inputs and reference outputs; alert on drift in CI.
- Log everything. Persist prompts, settings, model ID, and context artifacts for each run.
- Document safety rails. Record blocked terms and moderation policies so shifts are visible.
- Backstop with evaluation. Use spot checks and simple metrics (format accuracy, schema validity, length).
For API specifics and current capabilities, see the Anthropic docs.
Example you can copy: a five‑sentence fable pattern
Goal: lock the style, not the story. Keep creativity within a durable shape.
- System (versioned in Git): “You write compact five‑sentence fables. Tone: wry, modern, suitable for adults. Always include a one‑line moral starting with ‘Moral:’. No emojis.”
- User template: “Write a five‑sentence fable about {topic}. Keep it under 120 words.”
- Settings: temperature=0.3, top_p=0.9, pinned model snapshot, structured output with two fields: {“fable”: string, “moral”: string}.
Tip: Save 5–10 golden prompts (e.g., startup, healthcare, hiring) and commit their outputs. When a provider updates, rerun and diff. If structure holds and tone stays close, you’ve achieved practical permanence.
Governance for teams
- Require PRs for prompt changes; include before/after evals.
- Expose model ID, temperature, and seed in your UI so runs are auditable.
- Separate “creative mode” from “stable mode” with presets.
- Schedule monthly drift checks across golden prompts and data snapshots.
Takeaway
You can’t freeze an LLM forever, but you can make its behavior feel permanent. Pin versions, fix settings, constrain structure, and test for drift. That’s stability without strangling creativity.
Further reading: Simon Willison on making a creative pattern feel permanent.
If you found this useful, get more bite‑size, practical AI tips in your inbox. Subscribe to The AI Nuggets newsletter.
