The study tested 24 MLLMs using Raven's Progressive Matrices but found overall poor performance; the models struggled with visual interpretation and pattern recognition.
Research assistant Kian Ahrabian stated, "They were really bad. They couldn't get anything out of it," highlighting MLLM limitations in cognitive tasks designed for humans.
Results showed open-source models struggled more than closed-source models like GPT-4V, which used Chain of Thought prompting to improve visual reasoning abilities.
The challenge involved models applying patterns in visual puzzles, akin to human cognitive processes, indicating significant differences in reasoning capabilities.
Collection
[
|
...
]