Do large language models solve verbal analogies like children do?
Stevenson, Claire E., ter Veen, Mathilde, Choenni, Rochelle, van der Maas, Han L. J., Shutova, Ekaterina
–arXiv.org Artificial Intelligence
Analogy-making lies at the heart of human cognition. Adults solve analogies such as \textit{Horse belongs to stable like chicken belongs to ...?} by mapping relations (\textit{kept in}) and answering \textit{chicken coop}. In contrast, children often use association, e.g., answering \textit{egg}. This paper investigates whether large language models (LLMs) solve verbal analogies in A:B::C:? form using associations, similar to what children do. We use verbal analogies extracted from an online adaptive learning environment, where 14,002 7-12 year-olds from the Netherlands solved 622 analogies in Dutch. The six tested Dutch monolingual and multilingual LLMs performed around the same level as children, with MGPT performing worst, around the 7-year-old level, and XLM-V and GPT-3 the best, slightly above the 11-year-old level. However, when we control for associative processes this picture changes and each model's performance level drops 1-2 years. Further experiments demonstrate that associative processes often underlie correctly solved analogies. We conclude that the LLMs we tested indeed tend to solve verbal analogies by association with C like children do.
arXiv.org Artificial Intelligence
Oct-31-2023
- Country:
- North America > United States
- New York (0.04)
- Europe
- Sweden > Östergötland County
- Linköping (0.04)
- Netherlands > North Holland
- Amsterdam (0.04)
- Sweden > Östergötland County
- North America > United States
- Genre:
- Research Report > New Finding (1.00)
- Industry:
- Education > Educational Setting (0.46)
- Technology: