Large Language Models (LLMs) encounter significant challenges in continual learning due to catastrophic forgetting, where new information overwrites previously acquired knowledge. This limitation leads to substantial environmental and economic waste. In this study, we introduce the PMoE, Progressive Mixture of Experts with Asymmetric Transformer, which aims to minimize forgetting by utilizing an asymmetric design with shallow layers dedicated to general knowledge and deep layers for new knowledge. PMoE incorporates progressively added experts in deep layers and a router that allocates new knowledge to the appropriate experts efficiently. The router, positioned adjacent to the deep layers, utilizes deep features aggregating consolidated information. This enables the router to perform efficiently, allocating new knowledge to the appropriate experts, which progressively increase in the deep layers. Extensive experiments on TRACE datasets and general language understanding datasets demonstrate that the proposed PMoE outperforms previous state-of-the-art approaches.

This paper suggested constructing different binary data representation using different hash functions in the asymmetric theme to possibly reduce the encoding length. Encoding data as compactly as possible is no doubt important to theory and practice of approximate nearest neighbor search. A major concern of the asymmetric design is the consistency of the results. Therefore, by this design it is very possible to obtain inconsistent results such as u(x), v(y) u(y), v(x), which means that the learned hash functions CANNOT support a distance metric. And in the actual experiments, the authors used whether a certain point x is in the database to determine which hash function to use, which is ad-hoc and lacks any ground support.

In this paper, we propose a method to avoid "no-solution" situations of the control barrier function (CBF) for distributed collision avoidance in a multiagent autonomous robotic system (MARS). MARS, which is composed of distributed autonomous mobile robots, is expected to effectively perform cooperative tasks such as searching in a certain area. Therefore, collision avoidance must be considered when implementing MARS in the real world. The CBF is effective for solving collision-avoidance problems. However, in extreme conditions where many robots congregate at one location, the CBF constraints that ensure a safe distance between robots may be violated. We theoretically demonstrate that this problem can occur in certain situations, and introduce an asymmetric design for the inequality constraints of CBF. We asymmetrically decentralized inequality constraints with weight functions using the absolute speed of the robot so that other robots can take over the constraints of the robot in severe condition. We demonstrate the effectiveness of the proposed method in a two-dimensional situation wherein multiple robots congregate at one location. We implement the proposed method on real robots and the confirmed the effectiveness of this theory.

Recently, a variety of methods under the name of non-contrastive learning (like BYOL, SimSiam, SwAV, DINO) show that when equipped with some asymmetric architectural designs, aligning positive pairs alone is sufficient to attain good performance in self-supervised visual learning. Despite some understandings of some specific modules (like the predictor in BYOL), there is yet no unified theoretical understanding of how these seemingly different asymmetric designs can all avoid feature collapse, particularly considering methods that also work without the predictor (like DINO). In this work, we propose a unified theoretical understanding for existing variants of non-contrastive learning. Our theory named Rank Differential Mechanism (RDM) shows that all these asymmetric designs create a consistent rank difference in their dual-branch output features. This rank difference will provably lead to an improvement of effective dimensionality and alleviate either complete or dimensional feature collapse. Different from previous theories, our RDM theory is applicable to different asymmetric designs (with and without the predictor), and thus can serve as a unified understanding of existing non-contrastive learning methods. Besides, our RDM theory also provides practical guidelines for designing many new non-contrastive variants. We show that these variants indeed achieve comparable performance to existing methods on benchmark datasets, and some of them even outperform the baselines. Our code is available at https://github.com/PKU-ML/