No Need to Talk: Asynchronous Mixture of Language Models