Do Vision-Language Models Use Visual Context for Idiom Understanding? A Multimodal Evaluation of Idiom Detection
Idioms are difficult for models because their figurative meaning often differs from their literal word meaning. In multimodal settings, images could help models decide whether an idiom is used literally or figuratively. Inspired by AdMIRe 2, we reformulate the dataset as an idiom detection task in which a VLM receives a sentence and an image as context.
This project evaluates whether state-of-the-art vision-language models use visual context for idiom detection. Given a sentence containing an idiom and an accompanying image, the model must classify the expression as literal or figurative. Our results show that even large VLMs do not consistently benefit from image cues: images sometimes help, sometimes hurt, and often appear to be ignored. This suggests that current VLMs may rely more on textual priors than on genuine multimodal reasoning for idiom understanding.