Claim◆Article

The pelican benchmark's biggest limitation is that it says nothing about agentic tool calling and reliable tool use over long conversations, which is what matters most for today's models.

Willison points out the pelican test entirely misses agentic tool-calling ability, now the most important quality for evaluating models. ✦ AI generated

Simon Willison · Simon Willison's Weblog · 2026-07-16 · original ↗

Share card ↓Export .md Export .txt

The biggest limitation of the pelican is that it doesn't touch at all on the thing that matters most for today's model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.

Read full article ↗excerpt · fair-use quotation

✳ pelican-riding-a-bicycle

Related moments

Claim◆Video · 44:29 · 2m

No one has run a frontier model continuously for a long stretch (e.g. a full year) because each generation is superseded so quickly, so we may never actually know how intelligent these models truly are or could become.

Gavin Baker · BG2 Pod

Claim◆Article · 29 words

Muse Spark 1.1 delivers significant improvements over the original Muse Spark in agentic tool calling and computer use.

Meta · Simon Willison's Weblog

Claim◆Article · 31 words

The pelican benchmark's former correlation with actual model quality has broken down, since GLM-5.2's pelican outclasses those from GPT-5.6 and Claude Fable 5 despite GLM not being a frontier-class model.

Simon Willison · Simon Willison's Weblog