AI can write code, but it can’t own outcomes. In this excellent essay, Simon Willison lays out why developers still matter. Here’s the practical playbook: where AI helps, where it fails, and how to ship faster without breaking prod.
Why AI hasn’t replaced software engineers
- Specs are the hard part: LLMs generate code from prompts, but they don’t resolve conflicting requirements, trade-offs, or edge cases. Humans turn ambiguity into a shippable spec.
- Systems thinking beats snippets: Real work spans APIs, state, data models, and deployment. AI is strong at local code, weak at end-to-end architectural intent.
- Reliability and safety: Models hallucinate, overfit patterns, and miss non-functional needs (perf, cost, security). Production-grade code needs tests, SLOs, and accountability.
- Context is fragmented: Critical knowledge lives in tickets, Slack, ADRs, and tribal memory. LLMs rarely see the full picture without deliberate retrieval.
- Ownership matters: Shipping is maintenance, observability, and incident response. Tools assist; engineers are responsible.
What AI is genuinely good at today
- Boilerplate, scaffolds, and CRUD glue
- Translating between languages, frameworks, or SDK versions
- Regex, small utilities, and refactor templates
- Test skeletons and property-based test ideas
- Summarizing code, generating docstrings, and drafting migration guides
A practical workflow to use LLMs without breaking prod
- Start with a tight spec: Describe inputs, outputs, constraints, and examples. Paste relevant interfaces and schemas.
- Constrain the task: Ask for small, reviewable diffs. “Modify only function X; keep public API unchanged; include unit tests.”
- Adopt a plan-first prompt: “Propose a file-by-file plan first. Wait for approval. Then generate patches step by step.”
- Generate tests before code: Have the model propose edge cases and property tests, then implement to satisfy them.
- Ground the model in your repo: Provide code snippets, ADRs, and API contracts. Use retrieval or targeted context windows.
- Review like a security engineer: Demand explanations for changes, complexity, and potential failure modes. Never auto-commit.
Safeguards that catch AI mistakes
- Run in a sandbox with tight permissions; generate throwaway branches
- Static analysis, linters, and policy-as-code (e.g., supply-chain checks)
- Contract tests and golden files for critical paths
- Load and fuzz tests for anything touching parsing or untrusted input
- Require human sign-off for migrations, schema changes, and secrets
Metrics that keep you honest
- PR cycle time and review latency
- Escaped defects per KLOC and incident rate
- Test coverage on changed lines (diff coverage)
- Rework percentage (follow-up fix commits)
- Cost-to-serve deltas (CPU, memory, egress) post-change
Evidence: useful, not magical
Field studies show AI can speed up certain tasks. GitHub reported developers completed a task up to 55% faster with Copilot, mainly on boilerplate. Adoption is mainstream too—see Stack Overflow’s 2024 Developer Survey. But speed-ups don’t replace system design, testing, or accountability.
For a grounded perspective on why replacement hasn’t happened, read Simon Willison’s Why AI Hasn’t Replaced Software Engineers.
Takeaway
Treat LLMs like fast junior assistants: great at drafts and recall, unreliable as owners. Constrain tasks, test first, and make humans accountable for outcomes.
Like this? Get one practical AI nugget in your inbox each week. Subscribe to the newsletter.
