CoLM: Collaborative Large Models via A Client-Server Paradigm