The researchers introduced LLaVA-CoT to enhance multimodal reasoning capabilities by utilizing a structured, multistage approach, outperforming larger models like Gemini-1.5-pro and GPT-4o-mini.
The model relies on a systematic method where it does not produce direct responses but instead undergoes a four-stage reasoning process: summary, caption, reasoning, and conclusion.
To support this structured reasoning framework, a custom dataset, LLaVA-o1-100k, was created by employing GPT-4o to generate stage-wise responses, combining data from various VQA sources.
LLaVA-CoT addresses the frequent hallucinations and errors in visual language models by improving their understanding of reasoning stages, ensuring clearer task identification and structured thought processes.
Collection
[
|
...
]