MechanismArticle

You can run smevals by telling your coding agent to run uvx smevals docs to learn the tool, then build an eval suite, and run it against models with uvx smevals run.

Explains the quickstart workflow: uvx smevals docs to learn, build an eval suite, then run it against models. ✦ AI generated

Simon Willison · Simon Willison's Weblog · 2026-07-31 · original ↗

Share card ↓Export .md Export .txt

Tell your coding agent to run uvx smevals docs to learn the tool (this outputs the README) Then tell it to build you an eval suite Once you've created an eval - which takes the form of a directory with some YAML files - you can run it against models like this: uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6

Read full article ↗excerpt · fair-use quotation

✳ Claude Opus 4.6 ✳ GPT-5.5 ✳ smevals

Around this claim

In practice · 1

smevals is a new tool for running small eval suites across different model configurations and grading the results.Simon Willison · Simon Willison's Weblog · conf 95%

Context · 1

smevals is my third iteration on the idea of an evals approach and it feels right to me.Simon Willison · Simon Willison's Weblog · conf 60%

This moment responds to

extends → smevals separates runs from grading operations, and can output results as a static HTML report via the build command.Simon Willison · Simon Willison's Weblog