Article · 2026-07-16 · 6 moments

Kimi K3, and what we can still learn from the pelican benchmark

Chinese AI lab Moonshot AI announced Kimi K3 this morning, describing it as their "most capable model to date, with 2.8 trillion parameters". It's currently available via their website and API, but an open weight release is promised "by July 27, 2026". Moonshot are calling this the first "open 3T-class model" (I guess they're rounding 2.8 trillion up to 3 trillion), taking the crown from DeepSeek's 1.6T v4 Pro. Their self-reported benchmarks have K3 mostly beating Claude Opus 4.8 max and GPT-5. ✦ AI generated

Read original ↗

HOST Simon Willison

Claim

Despite being a flawed comparative benchmark, the pelican test remains valuable to Willison mainly as a forcing function that ensures he actually runs a prompt through each new model.

Willison argues the real value of the pelican test isn't as a rigorous benchmark but as a forcing function that gets him to actually try each new model.

transcript

Simon Willison: Firstly, it's a forcing function for actually trying the model. If I show you a pelican, that means I've managed to run a prompt through it. If the model has an official API I'll use that, if it's open weight... I'll try running it on my own machine.

Example

Kimi K3 currently offers only one reasoning effort level, 'max,' which makes it expensive to run: it burned 13,241 reasoning tokens to produce just 3,417 tokens of output, costing 25 cents for a single pelican SVG.

Running the pelican prompt through K3 showed the model only supports one, costly 'max' reasoning setting, spending far more tokens on reasoning than on the actual output.

transcript

Simon Willison: It only has one reasoning effort right now, "max" - and it shows. The model consumed 13,241 reasoning tokens to output 3,417 tokens of response. This is expensive - the pelican cost 25 cents!

Context

Kimi K3 is Moonshot AI's most capable model to date, a 2.8 trillion parameter model being called the first open 3T-class model, surpassing DeepSeek's 1.6T v4 Pro.

Moonshot AI announced Kimi K3, a 2.8T parameter model they're branding as the first 'open 3T-class model,' with an open weight release promised by July 27, 2026.

transcript

Simon Willison: Moonshot are calling this the first "open 3T-class model" (I guess they're rounding 2.8 trillion up to 3 trillion), taking the crown from DeepSeek's 1.6T v4 Pro.

extends · 2supports · 1

Data

Kimi K3's pricing of $3/million input and $15/million output tokens matches Anthropic's Claude Sonnet series and makes it the most expensive Chinese-lab model released to date, a large jump from K2.6's $0.95/$4.

K3's pricing is a sharp increase over Moonshot's earlier models and puts it in the same price bracket as Claude Sonnet, making it the priciest Chinese-lab release yet.

transcript

Simon Willison: The new model is notable for the pricing: $3/million input tokens and $15/million output tokens, putting it at the same level as Anthropic's Claude Sonnet series and making it the most expensive model released by a Chinese AI lab to date.

Claim◆

The pelican benchmark's biggest limitation is that it says nothing about agentic tool calling and reliable tool use over long conversations, which is what matters most for today's models.

Willison points out the pelican test entirely misses agentic tool-calling ability, now the most important quality for evaluating models.

transcript

Simon Willison: The biggest limitation of the pelican is that it doesn't touch at all on the thing that matters most for today's model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.

Claim◆

The pelican benchmark's former correlation with actual model quality has broken down, since GLM-5.2's pelican outclasses those from GPT-5.6 and Claude Fable 5 despite GLM not being a frontier-class model.

Willison says the pelican test's early correlation with model quality is now mostly gone, since a weaker model (GLM-5.2) draws a better pelican than top models.

transcript

Simon Willison: That connection has been mostly severed now. The GPT-5.6 and Claude Fable 5 pelicans are outclassed by GLM-5.2, and much as I love GLM I don't think that's a Fable-class model.

Highlight slides

Pelican Benchmark Loses Its Signal✦ from: The pelican benchmark's former correlation with actual model quality has broken down, since GLM-5.2's pelican outclasses those from GPT-5.6 and Claude Fable 5 despite GLM not being a frontier-class model.Pelican Test's Biggest Blind Spot✦ from: The pelican benchmark's biggest limitation is that it says nothing about agentic tool calling and reliable tool use over long conversations, which is what matters most for today's models.Willison's Take✦ from: The pelican benchmark's former correlation with actual model quality has broken down, since GLM-5.2's pelican outclasses those from GPT-5.6 and Claude Fable 5 despite GLM not being a frontier-class model.What Gets Missed✦ from: The pelican benchmark's biggest limitation is that it says nothing about agentic tool calling and reliable tool use over long conversations, which is what matters most for today's models.

Related episodes

Video · The Cognitive Revolution

Despite being a flawed comparative benchmark, the pelican test remains valuable to Willison mainly as a forcing function that ensures he actually runs a prompt through each new model.

Kimi K3 currently offers only one reasoning effort level, 'max,' which makes it expensive to run: it burned 13,241 reasoning tokens to produce just 3,417 tokens of output, costing 25 cents for a single pelican SVG.

Kimi K3 is Moonshot AI's most capable model to date, a 2.8 trillion parameter model being called the first open 3T-class model, surpassing DeepSeek's 1.6T v4 Pro.

Kimi K3's pricing of $3/million input and $15/million output tokens matches Anthropic's Claude Sonnet series and makes it the most expensive Chinese-lab model released to date, a large jump from K2.6's $0.95/$4.

The pelican benchmark's biggest limitation is that it says nothing about agentic tool calling and reliable tool use over long conversations, which is what matters most for today's models.

The pelican benchmark's former correlation with actual model quality has broken down, since GLM-5.2's pelican outclasses those from GPT-5.6 and Claude Fable 5 despite GLM not being a frontier-class model.

Fable's Back, AI Engineer Recap, & SambaNova

Zuckerberg Back on X Challenging Codex & Claude Code | SK Hynix’s $26BN IPO

LGTM, Ship It: The AI Code Review Problem