Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset

While recent NLP evaluation benchmark tasks test some aspects of human-imitative behavior (e.g., BIG-bench's'human-like behavior' tasks), few, if not none, examine creative problem solving abilities.