PredictionArticle

Kimi K3's benchmark story might be overstated unless validated on hidden or uncontaminated evals like LiveBench, and if the model 'thinks forever,' its real-world cost could end up less favorable than advertised.

Bindu Reddy cautioned that K3's impressive leaderboard results could be inflated by benchmark contamination and that its verbose 'thinking' behavior might erode its apparent cost advantage in practice. ✦ AI generated

Bindu Reddy · Latent Space · 2026-07-17 · original ↗

Share card ↓Export .md Export .txt

Bindu Reddy warned that K3's benchmark story might be overstated unless validated on hidden / uncontaminated evals like LiveBench, and argued that if the model "thinks forever," real cost could be less favorable

Read full article ↗excerpt · fair-use quotation

✳ Kimi K3 ✳ LiveBench ◉ Bindu Reddy

Around this claim

Mechanism · 1

Kimi K3 currently offers only one reasoning effort level, 'max,' which makes it expensive to run: it burned 13,241 reasoning tokens to produce just 3,417 tokens of output, costing 25 cents for a single pelican SVG.Simon Willison · Simon Willison's Weblog · conf 60%

Evidence · 1

Kimi K3's pricing of $3/million input and $15/million output tokens matches Anthropic's Claude Sonnet series and makes it the most expensive Chinese-lab model released to date, a large jump from K2.6's $0.95/$4.Simon Willison · Simon Willison's Weblog · conf 60%

This moment responds to

rebuts → Kimi K3 hit 1668 Elo on GDPval v2, 53% and #1 on AutomationBench-AA, and 1547 Elo on AA-Briefcase, at $0.94 cost per task and about 21% fewer output tokens than K2.6 across the full Intelligence Index run.Artificial Analysis · Latent Space