Simon Willison highlighted a notable price cut for Luna—another sign that AI inference is getting cheaper. Here’s what it means for your roadmap and how to act fast. Source: simonwillison.net.
Why this matters
- Competitive pressure: foundation model providers are cutting rates to win developer mindshare.
- Better efficiency: quantization, batching, and improved serving stacks are squeezing cost per token.
- Cloud economics: more capacity, lower marginal costs, and pass-through discounts are arriving downstream.
- Budget agility: pricing is now a variable you can optimize monthly—not a fixed line item.
What to do this week
- Audit token spend: break down prompts, outputs, embeddings, and RAG lookups. Turn on response streaming and set stop sequences to cut waste.
- Right-size models: route easy tasks to smaller/cheaper models; reserve premium models for high-value paths. Consider prompt distillation.
- Compress context: use retrieval filters, summaries, and structured tools to shrink context windows without losing accuracy.
- Batch and async: queue non-urgent jobs, batch embeddings/inference, and use scheduled windows to capture lower-cost capacity.
- Negotiate usage: ask vendors for committed-use discounts and overage protection; keep a second provider hot for leverage.
- Track hidden costs: embeddings, vector DB storage/egress, logging, and eval runs can rival inference—monitor them.
Benchmarks to watch
- Effective $/1K tokens (prompt + completion + embeddings)
- Cost per successful action (not just per request)
- P95 latency at target cost
- Throughput per dollar with and without batching
Helpful references
- Original note on Luna’s price change by Simon Willison: read it here.
- OpenAI pricing (for cross-checking market rates): openai.com/pricing.
Takeaway
Treat model pricing as dynamic. Set a 30-day rhythm to re-evaluate providers, re-route workloads, and renegotiate. Savings compound quickly.
Like nuggets like this? Subscribe for weekly, no-fluff AI tactics: theainuggets.com/newsletter.
