Researchers at SplxAI compared OpenAI's new reasoning model, o3-pro, against the multimodal GPT-4o to assess performance and reliability. The study found o3-pro to be significantly less efficient, using 7.3 times more tokens and costing 14 times more to operate while failing in 5.6 times more test cases. The findings indicate that o3-pro's complex reasoning mechanisms may lead to unnecessary complications, emphasizing the importance of not taking vendor claims at face value when selecting AI models.
Developers shouldn't take vendor claims as dogma and immediately go and replace their LLMs with the latest and greatest from a vendor.
The results underscored that o3-pro, while branded as advanced, is far less performant, reliable, and secure in comparison to GPT-4o.
Collection
[
|
...
]