Current AI agents remain far from reliable at long-horizon computer use, with the best configuration, Claude Opus 4.8 with maximum thinking, reaching only 20.6% binary accuracy on OSWORLD 2.0's multi-hour tasks, and performance dropping sharply as tasks lengthen.

OSWorld 2.0's creators find that even the strongest model setup only hits 20.6% binary accuracy on tasks averaging 1.6 hours of human effort, with agents struggling most on hidden-state recovery, tracking many items, and adapting to changing requirements. ✦ AI generated

OSWorld 2.0 researchers · Import AI · 2026-07-06 · original ↗

Our experiments show that current agents remain far from reliable computer use: the strongest setting, Claude Opus 4.8 with maximum thinking and batched tool calls, reaches only 20.6% binary accuracy and 54.8% partial-score accuracy. Performance drops sharply as tasks grow longer, and agents struggle most when they must recover hidden state, track many items, resolve conflicting information, or adapt to changing requirements.

Read full article ↗excerpt · fair-use quotation