Statistical Learning
Mixture of Experts Meets Prompt-Based Continual Learning Minh Le3 An Nguyen 2 Huy Nguyen 1 Trang Nguyen
However, current AI models often fail to retain this ability.
Wasserstein Gradient Boosting: A Framework for Distribution-Valued Supervised Learning
It solves distribution-valued supervised learning, where the output values of the training dataset are probability distributions.
Derivatives of Stochastic Gradient Descent in parametric optimization
We consider stochastic optimization problems where the objective depends on some parameter, as commonly found in hyperparameter optimization for instance. We investigate the behavior of the derivatives of the iterates of Stochastic Gradient Descent (SGD) with respect to that parameter and show that they are driven by an inexact SGD recursion on a different objective function, perturbed by the convergence of the original SGD. This enables us to establish that the derivatives of SGD converge to the derivative of the solution mapping in terms of mean squared error whenever the objective is strongly convex.
Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse
Neural networks are often trained on multiple tasks, either simultaneously (multi-task learning, MTL) or sequentially (pretraining and subsequent finetuning, PT+FT). In particular, it is common practice to pretrain neural networks on a large auxiliary task before finetuning on a downstream task with fewer samples. Despite the prevalence of this approach, the inductive biases that arise from learning multiple tasks are poorly characterized. In this work, we address this gap.
Improving Generalization and Convergence by Enhancing Implicit Regularization Mingze Wang 1,3, Jinbo Wang 1, 3 Haotian He1,3 Zilin Wang 1
We show that IRE can be practically incorporated with generic base optimizers without introducing significant computational overload.
Treeffuser: Probabilistic Predictions via Conditional Diffusions with Gradient-Boosted Trees Nicolas Beltran-V elez
Probabilistic prediction aims to compute predictive distributions rather than single point predictions. These distributions enable practitioners to quantify uncertainty, compute risk, and detect outliers. However, most probabilistic methods assume parametric responses, such as Gaussian or Poisson distributions.
Deep Support Vectors
Deep learning has achieved tremendous success.
Deep Bayesian Active Learning for Preference Modeling in Large Language Models
We address this by proposing the B ayesian A ctive L earner for P reference M odeling (BAL-PM), a novel stochastic acquisition policy that not only targets points of high epistemic uncertainty according to the preference model but also seeks to maximize the entropy of the acquired prompt distribution in the feature space spanned by the employed LLM.