Unleashing the Power of LLMs as Multi-Modal Encoders for Text and Graph-Structured Data