Simon Willison just highlighted Nativ, a new entrant in the on-device AI runtime space (link). Whether or not you adopt it, here’s a practical way to evaluate any flashy LLM runtime before it lands in your stack.
On-device models promise lower latency, privacy, and better cost control—but hype can mislead. Use this checklist to separate signal from noise.
A fast checklist to vet any new LLM runtime
- Hardware fit and formats: Does it support the chips you have (CPU/GPU/NPU) and common model formats (e.g., GGUF, ONNX)? Are quantizations like INT8/INT4 stable?
- Reproducible benchmarks: Provide exact prompts, seeds, tokenizer, and context lengths. Report time-to-first-token, tokens/sec, peak memory, and power draw—not just a single number.
- Model coverage and licensing: Which model families are validated? Are commercial use and redistribution allowed? Any EULA traps?
- Quality and safety: Compare quality vs. baselines using public evals. Test JSON/tool-call accuracy and refusal behavior, not only speed.
- Developer ergonomics: One-liner install, OpenAI-compatible endpoints, streaming, batching, SDKs, and clear examples.
- Observability: Logs, metrics, and traces. Prometheus/OpenTelemetry support, basic profiling, and actionable error messages.
- Deployment realities: Startup time, cold-start behavior, CPU-only performance, container size, and offline mode.
- Governance and trust: Active maintainers, roadmap, security policy/CVE process, and semantic versioning with changelogs.
Run a 30-minute smoke test
- Pick a small, permissive model (e.g., Llama 3.1 8B Instruct) in two formats (GGUF and ONNX if available).
- Establish a baseline with a known runtime like llama.cpp or Hugging Face TGI.
- Replicate with the new runtime using identical prompts, sampling params, tokenizer, and context length; fix a random seed.
- Measure time-to-first-token, tokens/sec, peak RAM/VRAM, CPU/GPU utilization, and power; save full command lines for reproducibility.
- Exercise JSON mode and tool/function calling if supported; verify structured outputs for correctness.
- Stress long prompts near max context to observe degradation, latency spikes, or crashes.
- Score the developer loop: install time, docs quality, errors, and compatibility with your app’s API surface.
When to prefer on-device vs. cloud
- Choose on-device for low-latency UX, privacy/data locality, offline reliability, and predictable unit economics.
- Choose cloud for very long context, multi-tenant throughput, access to frontier models, and fast upgrades without shipping binaries.
Sources
Read Simon Willison’s note on Nativ: simonwillison.net/2026/Jul/21/nativ. For baselines, see llama.cpp and Hugging Face TGI.
Takeaway
New runtimes will keep arriving. Trust reproducible tests and dev ergonomics—not headline tokens/sec—then adopt only if it cuts latency, cost, or complexity in your stack.
Get more bite-size, practical AI insights in your inbox—subscribe to The AI Nuggets.
