ClaimArticle

Despite being a flawed comparative benchmark, the pelican test remains valuable to Willison mainly as a forcing function that ensures he actually runs a prompt through each new model.

Willison argues the real value of the pelican test isn't as a rigorous benchmark but as a forcing function that gets him to actually try each new model. ✦ AI generated

Simon Willison · Simon Willison's Weblog · 2026-07-16 · original ↗

Share card ↓Export .md Export .txt

Firstly, it's a forcing function for actually trying the model. If I show you a pelican, that means I've managed to run a prompt through it. If the model has an official API I'll use that, if it's open weight... I'll try running it on my own machine.

Read full article ↗excerpt · fair-use quotation

✳ LLM CLI ✳ pelican-riding-a-bicycle

Related moments

Claim◆Article · 31 words

The pelican benchmark's former correlation with actual model quality has broken down, since GLM-5.2's pelican outclasses those from GPT-5.6 and Claude Fable 5 despite GLM not being a frontier-class model.

Simon Willison · Simon Willison's Weblog

Claim◆Article · 37 words

The pelican benchmark's biggest limitation is that it says nothing about agentic tool calling and reliable tool use over long conversations, which is what matters most for today's models.

Simon Willison · Simon Willison's Weblog