Can GNN be Good Adapter for LLMs?