smevals - a small eval suite for evaluating models, prompts, and harnesses I've been working with Jesse Vincent's Prime Radiant applied AI research lab building out this evals framework to help answer questions about the capabilities of different models. The result is smevals, a new tool for running small eval suites across different model configurations and grading the results. This blog entry describes the tool in detail. Here's the 10 second version: Tell your coding agent to run uvx smev ✦ AI generated
Introduces smevals, a tool for running small eval suites across different model configurations and grading results.
Simon Willison: The result is smevals, a new tool for running small eval suites across different model configurations and grading the results.
provides context · 1
The author states this is their third iteration on an evals approach and it finally feels right.
Simon Willison: I've been trying to figure out an approach I like for evals for several years now. smevals is my third iteration on the idea and it feels right to me.
extends · 1
Describes the separation of run and grade operations, and the ability to generate static HTML reports from results.
Simon Willison: Runs are treated separately from grading operations - you can grade your runs (against your defined set of checks) using: uvx smevals grade path-to-eval/ Then you can run a localhost web server to explore the results: uvx smevals serve path-to-eval/ Or run the smevals build command to build that report as static HTML, which you can then host anywhere.
extends · 2provides context · 1
Explains the quickstart workflow: uvx smevals docs to learn, build an eval suite, then run it against models.
Simon Willison: Tell your coding agent to run uvx smevals docs to learn the tool (this outputs the README) Then tell it to build you an eval suite Once you've created an eval - which takes the form of a directory with some YAML files - you can run it against models like this: uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6
gives example · 1provides context · 1