CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks

Abstract--The increasing demand for intelligent mobile applications has made multi-agent collaboration with Transformer-based large language models (LLMs) essential in mobile edge computing (MEC) networks. However, training LLMs in such environments remains challenging due to heavy computation, high end-to-end latency, and limited model generalization. We introduce CollaPipe, a hybrid distributed learning framework that integrates collaborative pipeline parallelism with federated aggregation to support self-evolving intelligent networks. In Col-laPipe, the encoder part is adaptively partitioned into variable-sized segments and deployed across mobile devices for pipeline-parallel training, while the decoder is deployed on edge servers to handle generative tasks. Then we perform global model update via federated aggregation. T o enhance training efficiency, we formulate a joint optimization problem that adaptively allocates model segments, micro-batches, bandwidth, and transmission power . We derive and use a closed-form convergence bound to design an Dynamic Segment Scheduling and Resource Allocation (DSSDA) algorithm based on Lyapunov optimization, ensuring system stability under long-term constraints. Extensive experiments on downstream tasks with Transformer and BERT models show that CollaPipe improves computation efficiency by up to 15.09%, reduces end-to-end latency by at least 48.98%, and cuts single device memory usage by more than half, enabling online learning in heterogeneous and dynamic communication environments. With the rapid development of artificial intelligence generated content (AIGC) technologies in mobile Internet of Things (IoT), AI agent systems powered by large language models (LLMs) are emerging as a critical enabler for next-generation intelligent applications in mobile edge computing (MEC) networks [1]-[3]. Jiewei Chen, Shaoyong Guo, and Xuesong Qiu are with the State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China (e-mail: {chenjiewei, syguo, xsqiu}@bupt.edu.cn). Xiumei Deng is with the Singapore University of Technology and Design, Singapore (e-mail: xiumei_deng@sutd.edu.sg). Ze-hui Xiong is with the School of Electronics, Electrical Engineering and Computer Science, Queen's University Belfast, United Kingdom (e-mail: z.xiong@qub.ac.uk).