Agent Benchmarks Paint a Grim Picture: 37% Success Rate on Enterprise Tasks, and RL Training May Be Making Agents Worse
New benchmarks from ServiceNow and ARC-AGI expose how far agents are from production readiness, while CUHK researchers find that reinforcement learning can teach agents to actively avoid gathering evidence.
Two new benchmarks released this week delivered a cold shower to anyone assuming AI agents are nearly ready for unsupervised enterprise deployment. ServiceNow's EnterpriseOps-Gym, a suite of 1,150 real-world IT operations tasks, found that the best-performing models achieved a 37% success rate — meaning they fail nearly two out of three times on the kinds of tasks enterprises most want to automate.
Unlock the full briefing
Get every story in today's briefing, the full archive, and the daily AI intelligence brief.
All stories today
Full archive
Daily brief
Cancel anytime. Payments powered by Stripe.