Product Launches

OpenAI agents now handle 46% of 2-4 hour research tasks, up from 23% in March

Share
OpenAI agents now handle 46% of 2-4 hour research tasks, up from 23% in March

OpenAI's AI agents have significantly improved their task completion rates, now handling 46% of research tasks that would take humans 2-4 hours, up from 23% in March. This progress highlights the growing role of AI in accelerating research and development.

TL;DR

  • OpenAI's AI agents have doubled their success rate in completing research tasks, now handling 46% of tasks that take humans 2-4 hours, up from 23% in March.
  • The success rate for more complex tasks (4-8 hours) also rose from 13% to 35% over the same period.
  • The trend suggests AI could dramatically speed up AI development, but questions remain about the quality and reliability of agent-generated outputs.

What happened

In a November 18 report, OpenAI revealed that its AI agents now complete nearly half of the research tasks delegated to them, which would typically take a human researcher 2-4 hours. This success rate has risen from 23% in March to 46% in October, according to The Rundown. For more complex tasks requiring 4-8 hours of human work, the success rate increased from 13% to 35% over the same period.

OpenAI valued the median agent usage at $2,400 per researcher per day in October, based on API list prices. A survey of 101 researchers found that the median respondent would hypothetically trade 40% of their team's headcount for three months of access to frontier agents. Daily runtime averaged 5.7 agent workdays per researcher, an elevenfold increase from previous measurements. More than two-thirds of researchers averaged daily peaks of at least four agents running simultaneously.

Why it matters

The improvement in AI agent performance suggests that AI could significantly accelerate the pace of AI development. More reliable agents could shorten the time between an idea, an experiment, and a useful result. If these results improve the next generation of models, researchers would then have better agents to help build the subsequent generation.

However, the trend has a caveat. OpenAI's task mix changed, so stronger models and better choices about what to delegate could both explain the improvement. Researchers still criticized agents' research judgment and complained about 'AI slop.' They credited humans with the sampled ideas behind GPT-6 Astra, while saying Codex helped generate key pretraining insights for GPT-6.2 Astra. One researcher expressed 'no confidence' that their edge in ideas would last.

Key facts

  • OpenAI's AI agents now complete 46% of 2-4 hour research tasks, up from 23% in March.
  • Success rate for 4-8 hour tasks rose from 13% to 35% over the same period.
  • Median agent usage valued at $2,400 per researcher per day in October.
  • Survey of 101 researchers found median respondent would trade 40% of team's headcount for three months of access to frontier agents.
  • Daily runtime averaged 5.7 agent workdays per researcher, an elevenfold increase.
  • More than two-thirds of researchers averaged daily peaks of at least four agents running simultaneously.
  • Anthropic's Claude led 26% of its AI research and development work in August, up from under 1% in February.
  • METR assessment suggests AI assistance may already be speeding up development at 1.9 times its historical rate.

Context

The trend of AI agents handling more research tasks is part of a broader movement towards automating and accelerating AI development. OpenAI's recent math work, including formal versions of many proofs and summaries of its reasoning, gives researchers something to scrutinize. However, publication alone does not establish that every result holds up, and labs may need to devote more time to replication and review as agents generate more candidate improvements.

Anthropic's assessment of Claude's role in its research and development work highlights the growing integration of AI agents in the AI development process. The external assessment by METR suggests that AI assistance may already be speeding up development, but the explanation remains uncertain due to the lack of a controlled experiment and redacted evidence.

Topics

Related coverage

Join the discussion

Have a take on this story? Weigh in with our community on Facebook.

💬 Discuss on Facebook →