first-hand human knowledge

Attention is all that matters.

We read the podcasts, essays and interviews — and hand you the arguments: who claims what, who rebuts, and the original voice one click away.

Banning general-purpose computation in favor of specialized anal AI agents' success rate on end-to-end, economically valuable fre Slowing or banning the open-source AI model ecosystem is an effe Frontier AI agents' reliability on long-horizon, multi-step, mul

Popular moments

This week This month All time

all topics benchmarks · 17 claude-code · 9 automation · 7 sqlite-utils · 7 policy · 7 agents · 7

DataArticle

On a new independent AutomationBench leaderboard spanning 657 tasks across 40 simulated SaaS apps, Claude Fable 5 leads with 48.6% accuracy, narrowly ahead of Opus 4.8 at 48.5%, while every model tested still breaks business-rule guardrails at some rate.

Artificial Analysis's independent AutomationBench-AA leaderboard shows Claude Fable 5 narrowly leading over Opus 4.8, with Gemini and GPT-5.5 further behind, and notes every model still violates business-rule guardrails.

Artificial Analysis · Latent Space

DataArticle

Current AI agents remain far from reliable at long-horizon computer use, with the best configuration, Claude Opus 4.8 with maximum thinking, reaching only 20.6% binary accuracy on OSWORLD 2.0's multi-hour tasks, and performance dropping sharply as tasks lengthen.

OSWorld 2.0's creators find that even the strongest model setup only hits 20.6% binary accuracy on tasks averaging 1.6 hours of human effort, with agents struggling most on hidden-state recovery, tracking many items, and adapting to changing requirements.

OSWorld 2.0 researchers · Import AI

DataArticle

Current frontier AI agents remain far from reliable at long-horizon, multi-step computer-use tasks, with even the strongest configuration (Claude Opus 4.8) scoring only 20.6% binary accuracy on OSWorld 2.0, struggling especially with hidden-state recovery, tracking many items, and conflicting information.

OSWorld 2.0, a benchmark of 108 long-horizon computer-use tasks averaging 1.6 hours for a human to complete, shows even top models like Claude Opus 4.8 achieving only 20.6% binary accuracy, though rapid gains are expected as happened with OSWorld 1.0.

OSWorld 2.0 paper authors · Import AI

DataArticle

Current frontier AI agents remain far from reliable at long-horizon, multi-program computer-use tasks, with the best-performing setup reaching only 20.6% binary accuracy on OSWORLD 2.0, though rapid improvement is likely given how quickly scores rose on OSWORLD 1.0.

OSWorld 2.0, a benchmark of multi-hour, multi-program computer-use tasks, shows even Claude Opus 4.8 with maximum thinking only reaches 20.6% binary accuracy, though the field expects a rapid ramp similar to OSWORLD 1.0's rise from ~30% to ~75%.

OSWorld 2.0 researchers · Import AIExtends · 1 Supports · 1