OpenAI’s o3 model might be costlier to run than originally estimated
When OpenAi revealed his O3 “reasoning” AI model in December, the company collaborated with the makers of ARC-Agi, a benchmark that was designed to test very capable AI, to present the possibilities of O3. Months later the results have been revised and they now look a little less impressive than initially.
Last week the ARC Prize Foundation, which maintains and manages ARC-Agi, updated estimated calculation costs for O3. The organization originally estimated that the best performing configuration of O3 IT tested, O3 High, cost around $ 3,000 to solve a single ARC-AGI problem. Now the Arc Prize Foundation thinks that the costs are much higher – possibly about $ 30,000 per task.
The revision is remarkable because it illustrates how expensive the most advanced AI models of today can be for certain tasks, at least early. OpenAi has not yet priced the O3 – or even released. But the Arc Prize Foundation believes that the O1-Pro model prices of OpenAI are a reasonable proxy.
For context, O1-Pro is the most expensive model in OpenAi so far.
“We believe that O1-Pro is a closer comparison of real O3 costs … because of the amount of test-time compute,” said Mike Knoop, one of the co-founders of the ARC Prize Foundation, to WAN. “But this is still a proxy, and we have labeled O3 as a preview on our leaderboard to display the uncertainty until the official prices have been announced.”
A high price for O3 High would not be excluded, given the amount of computer sources that the model reportedly uses. According to the ARC Prize Foundation, O3 High 172x used more computing than O3 Low, the lowest computer configuration of O3, to tackle ARC-Agi.
Moreover, rumors are already flying about pricey plans that OpenAi is considering introducing company customers. At the beginning of March, the information reported that the company may intend to charge up to $ 20,000 per month for specialized AI “agents”, such as a software developer agent.
Some can claim that even the most expensive models in OpenAI will cost well under what a typical human contractor or employee would recommend. But as an AI researcher Toby Ord Be on a message on XThe models may not be that efficient. For example, O3 High needed 1,024 attempts with every task in ARC-Agi to achieve its best score.
