With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models

Recently, Large Language Models (LLMs) and Vision Language Models (VLMs) have demonstrated aptitude as potential substitutes for human participants in experiments testing psycholinguistic phenomena. However, an understudied question is to what extent models that only have access to vision and text modalities are able to implicitly understand sound-based phenomena via abstract reasoning from orthography and imagery alone. To investigate this, we analyse the ability of VLMs and LLMs to demonstrate sound symbolism (i.e., to recognise a non-arbitrary link between sounds and concepts) as well as their ability to "hear" via the interplay of the language and vision modules of open and closed-source multimodal models. We perform multiple experiments, including replicating the classic Kiki-Bouba and Mil-Mal shape and magnitude symbolism tasks and comparing human judgements of linguistic iconicity with that of LLMs. Our results show that VLMs demonstrate varying levels of agreement with human labels, Figure 1: Illustration of the 3 main experiments we and more task information may be required perform. Firstly, Shape Symbolism is a binary choice for VLMs versus their human counterparts for between two pseudowords to best describe an object that is in silico experimentation. We additionally see spiky or rounded. Magnitude Symbolism involves a binary through higher maximum agreement levels that choice between two pseudowords to best describe an object Magnitude Symbolism is an easier pattern for that is small or large. Finally, Iconicity involves rating VLMs to identify than Shape Symbolism, and the perceived iconicity of words, or how much their written/phonetic that an understanding of linguistic iconicity is form is representative of what they describe.