Are AI labs pelicanmaxxing—optimizing for flashy demos over real, reproducible performance? Simon Willison raises the alarm, and it’s a useful lens for anyone evaluating new model launches.
Read his take here: Are AI labs pelicanmaxxing?.
What is “pelicanmaxxing”?
A shorthand for demo theater: cherry-picked prompts, heavy hand-holding, hidden constraints, and selective benchmarks that make models look magical.
The cure is evaluation hygiene—transparent methods, reproducible tests, and apples-to-apples baselines.
Red flags in AI lab demos
- No raw outputs or seeds shared—only glossy highlight reels.
- Unclear or proprietary evaluation sets; moving goalposts between versions.
- Hidden helpers: tool use, retrieval, or human intervention not disclosed.
- Selective baselines—comparing against weaker settings or outdated models.
- Non-reproducible prompts, undocumented system instructions, or temperature tricks.
- Safety and robustness results summarized without methodology or failure cases.
We’ve seen this before: edited or staged videos can inflate perceived capability. Example: reporting found Google’s early Gemini demo was significantly edited (The Verge).
Questions to ask before you believe the slide
- Reproducibility: Can I run the exact prompts with fixed seeds? Are system prompts, tools, and context windows disclosed?
- Baselines: What are the strongest comparable baselines (same date and settings)?
- Generalization: How does it handle slightly altered or out-of-distribution inputs?
- Costs and latency: What’s the throughput, token price, and hardware footprint?
- Safety: Are jailbreak resistance and harmful output rates measured, with methods and raw failures shared?
- Versioning: Are model versions pinned and evals timestamped for comparability?
Better yardsticks you can check today
- LMSYS Chatbot Arena: community head-to-head rankings and Arena Hard evals.
- Stanford HELM: transparent, multi-metric evaluations across realistic scenarios.
- MLCommons: standardized ML benchmarking initiatives and reports.
Takeaway
Don’t judge a model by its sizzle reel. Demand raw outputs, fixed seeds, strong baselines, and transparent methods. If you can’t reproduce it, don’t trust it.
Like this? Get one practical AI nugget in your inbox each week—subscribe to our newsletter: theainuggets.com/newsletter.
