ATRIUM search → argument graph

Claim◆Article

smevals is a new tool for running small eval suites across different model configurations and grading the results.

Introduces smevals, a tool for running small eval suites across different model configurations and grading results. ✦ AI generated

Simon Willison · Simon Willison's Weblog · 2026-07-31 · original ↗

Share card ↓Export .md Export .txt

The result is smevals, a new tool for running small eval suites across different model configurations and grading the results.

Read full article ↗excerpt · fair-use quotation

▦ Prime Radiant ✳ smevals ◉ Jesse Vincent

Around this claim

Context · 1

For research product managers at Anthropic, evals are the new PRDs — the primary artifact for defining user value has shifted from documents describing what to build to reproducible tests that measure whether the model has improved on specific user pain points.Dianne Penn · Lenny's Podcast · conf 70%

This moment responds to

gives example → You can run smevals by telling your coding agent to run uvx smevals docs to learn the tool, then build an eval suite, and run it against models with uvx smevals run.Simon Willison · Simon Willison's Weblog extends → smevals separates runs from grading operations, and can output results as a static HTML report via the build command.Simon Willison · Simon Willison's Weblog extends → smevals is my third iteration on the idea of an evals approach and it feels right to me.Simon Willison · Simon Willison's Weblog