On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions

In recent years, the integration of mixture-of-experts (MoE) within large-scale foundation models has markedly advanced the machine learning field [24, 11, 53, 73, 41]. MoE architectures, known for their ability to efficiently handle diverse and complex datasets, have facilitated significant improvements in model performance without a proportional increase in computational demand. They address bottlenecks associated with traditional deep learning architectures by dynamically allocating resources to parts of the model for which they are most relevant [67, 56]. The Hierarchical Mixture of Experts (HMoE) model [12] is a special type of MoE architecture that is characterized by a layered structure of decision modules and expert networks that operate in tandem to refine decision-making at each level, optimizing the allocation of computational resources and enhancing specialization for complex tasks. Unlike the standard MoE, which typically involves a single gating network directing inputs to various expert networks, HMoE introduces multiple layers of gating mechanisms and experts. This hierarchical design divides the problem space recursively, allowing different experts to specialize in subspaces of the input space, leading to enhanced flexibility and model generalization [25, 5]. Figure 1 compares HMoE and standard MoE in processing multimodal input data. The hierarchical structure of HMoE makes it particularly effective at handling complex inputs, such as data that can be divided into semantically meaningful subgroups.