Simon Willison’s latest writing on LLM tooling spotlights a practical path: treat AI like any other dependency—script it, log it, and test it. Here’s how to turn his “llm” CLI into a repeatable workflow your team can trust. Source: simonwillison.net.
What is “llm” and why devs use it
llm is a lightweight command-line tool that lets you call multiple LLM providers, store prompts/responses, and script AI tasks like any UNIX command. It’s fast to adopt, easy to version, and ideal for CI and data pipelines.
Quickstart: a reproducible CLI workflow
- Install:
pipx install llm(orpip install llm). Add provider plugins as needed, e.g.,llm install llm-openai. - Configure: set API keys as env vars (e.g.,
export OPENAI_API_KEY=...) and pin models in a.envor CI secret store. - Run a prompt:
llm -m gpt-4o "Summarize our release notes in 100 words". Pipe outputs like any CLI:cat notes.md | llm -m gpt-4o. - Make prompts reusable: save named prompts with
llm prompts set release_summary 'Summarize in 100 words, bullet points, neutral tone.'Then call withllm -p release_summary. - Log everything: use
llm logsto inspect prompt/response history. Store the SQLite file in your project for auditability. - Automate: wrap calls in
maketargets or shell scripts so teammates get the same results via one command.
Make outputs reliable
- Constrain formats: request JSON and validate it. Example: “Return a JSON object with fields title, summary, risk_level.” Fail loudly if parsing breaks.
- Tune determinism: lower temperature for consistency; keep prompts short, explicit, and include examples.
- Test like code: add snapshot tests for key prompts; for nondeterministic outputs, assert schema/keywords instead of exact text.
- Govern models: pin model names and versions; record provider, model, and parameters in logs for reproducibility.
Integrate with your stack
- Data pipelines:
csvcut | llm | jqlets you enrich datasets inline. Use chunking to stay within token limits. - Docs & changelogs: generate drafts from PR titles or commits:
git log --oneline | llm -p release_summary. - CI checks: run policy or style checks via
llmand fail the build on schema mismatch or high-risk flags. - Pre-commit hooks: standardize prompts across repos so tone and structure stay consistent.
Risks and safeguards
- Prompt injection: sanitize and scope inputs; avoid blindly following tool-use instructions from untrusted text. See the OWASP LLM Top 10.
- PII & secrets: route sensitive data to models with appropriate data handling policies; prefer server-side keys; redact logs.
- Cost drift: monitor token usage and set hard caps per job; cache results for repeated prompts.
- Model updates: major provider changes can shift outputs—lock versions and re-run snapshots on upgrades.
Resources
- Simon Willison on LLM tooling: blog post
- GitHub: simonw/llm
- Security: OWASP LLM Top 10
Takeaway
If AI is part of your product, treat prompts as code. With “llm” you can version, log, and test them—so features are reproducible, auditable, and shippable.
Get more practical AI tips in your inbox. Subscribe to our free newsletter: theainuggets.com/newsletter.
