Statistical Learning
Mixture of Experts Meets Prompt-Based Continual Learning Minh Le3 An Nguyen 2 Huy Nguyen 1 Trang Nguyen
However, current AI models often fail to retain this ability.
Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse
Neural networks are often trained on multiple tasks, either simultaneously (multi-task learning, MTL) or sequentially (pretraining and subsequent finetuning, PT+FT). In particular, it is common practice to pretrain neural networks on a large auxiliary task before finetuning on a downstream task with fewer samples. Despite the prevalence of this approach, the inductive biases that arise from learning multiple tasks are poorly characterized. In this work, we address this gap.
Improving Generalization and Convergence by Enhancing Implicit Regularization Mingze Wang 1,3, Jinbo Wang 1, 3 Haotian He1,3 Zilin Wang 1
We show that IRE can be practically incorporated with generic base optimizers without introducing significant computational overload.
Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
In particular, it aggregates multiple sub-models called experts based on a gating network. Here, experts can be formulated as neural networks, and they specialize in different aspects of the data.
Treeffuser: Probabilistic Predictions via Conditional Diffusions with Gradient-Boosted Trees Nicolas Beltran-V elez
Probabilistic prediction aims to compute predictive distributions rather than single point predictions. These distributions enable practitioners to quantify uncertainty, compute risk, and detect outliers. However, most probabilistic methods assume parametric responses, such as Gaussian or Poisson distributions.
