The article discusses the limitations of large Vision-Language models (VLMs) in understanding figurative language, which includes metaphors, similes, and humor. To address these challenges, the authors propose the V-FLUTE dataset, designed for explainable visual entailment tasks. This dataset includes over 6,000 examples across various figurative phenomena, and it facilitates human-AI collaboration to improve model evaluation. Through automatic and human assessments, the study highlights current VLMs' deficiencies in interpreting figurative language, stressing the importance of developing new approaches to enhance their reasoning capabilities.
Large Vision-Language models have shown strong reasoning capabilities but struggle with understanding figurative language, which relies on implicit meanings, presenting new challenges for AI.
We propose a new task for visual figurative language understanding that requires models to predict entailment between images and claims while providing textual justifications.
The V-FLUTE dataset comprises 6,027 instances of diverse multimodal figurative phenomena such as metaphors and sarcasm, which are crucial for improving AI's interpretation of nuanced communication.
Overall evaluation of current Vision-Language models indicates they are limited in their ability to handle figurative language, highlighting the need for new datasets and methods for improvement.
Collection
[
|
...
]