The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs