Continual Learning with Pre-trained Models holds great promise for efficient adaptation across sequential tasks. However, most existing approaches freeze PTMs and rely on auxiliary modules like prompts or adapters, limiting model plasticity and leading to suboptimal generalization when facing significant distribution shifts. While full fine-tuning can improve adaptability, it risks disrupting crucial pre-trained knowledge. In this paper, we propose Mutual Information-guided Sparse Tuning (MIST), a plug-and-play method that selectively updates a small subset of PTM parameters, less than 5%, based on sensitivity to mutual information objectives. MIST enables effective task-specific adaptation while preserving generalization. To further reduce interference, we introduce strong sparsity regularization by randomly dropping gradients during tuning, resulting in fewer than 0.5% of parameters being updated per step. Applied before standard freeze-based methods, MIST consistently boosts performance across diverse continual learning benchmarks. Experiments show that integrating our method into multiple baselines yields significant performance gains. Our code is available at https://github.com/zhwhu/MIST.

When language model (LM) users aim to improve the quality of its generations, it is crucial to specify concrete behavioral attributes that the model should strive to reflect. However, curating such principles across many domains, even non-exhaustively, requires a labor-intensive annotation process. To automate this process, we propose eliciting these latent attributes that guide model reasoning toward human-preferred responses by explicitly modeling them in a self-correction setting. Our approach mines new principles from the LM itself and compresses the discovered elements to an interpretable set via clustering. Specifically, we employ a form of posterior-regularized Monte Carlo Expectation-Maximization to both identify a condensed set of the most effective latent principles and teach the LM to strategically invoke them in order to intrinsically refine its responses. We demonstrate that bootstrapping our algorithm over multiple iterations enables smaller language models (7-8B parameters) to self-improve, achieving +8-10% in AlpacaEval win-rate, an average of +0.3 on MT-Bench, and +19-23% in principle-following win-rate on IFEval. We also show that clustering the principles yields interpretable and diverse model-generated constitutions while retaining model performance. The gains that our method achieves highlight the potential of automated, principle-driven post-training recipes toward continual self-improvement.

We propose a nonparametric density estimator based on the Gaussian process (GP) and derive three novel closed form learning algorithms based on Fisher divergence (FD) score matching. The density estimator is formed by multiplying a base multivariate normal distribution with an exponentiated GP refinement, and so we refer to it as a GP-tilted nonparametric density. By representing the GP part of the score as a linear function using the random Fourier feature (RFF) approximation, we show that optimization can be solved in closed form for the three FD-based objectives considered. This includes the basic and noise conditional versions of the Fisher divergence, as well as an alternative to noise conditional FD models based on variational inference (VI) that we propose in this paper. For this novel learning approach, we propose an ELBO-like optimization to approximate the posterior distribution, with which we then derive a Fisher variational predictive distribution. The RFF representation of the GP, which is functionally equivalent to a single layer neural network score model with cosine activation, provides a useful linear representation of the GP for which all expectations can be solved. The Gaussian base distribution also helps with tractability of the VI approximation and ensures that our proposed density is well-defined. We demonstrate our three learning algorithms, as well as a MAP baseline algorithm, on several low dimensional density estimation problems. The closed form nature of the learning problem removes the reliance on iterative learning algorithms, making this technique particularly well-suited to big data sets, since only sufficient statistics collected from a single pass through the data is needed.

Neural Information Processing Systems http://nips.cc/

Toys, crafts, lab kits, and more for the young loved ones in your life. In theory, buying gifts for children is a snap. If they're old enough to talk, but not old enough to ignore you completely, they will likely tell you what they want. And, if your kids run in the same kinds of circles as mine, they all seem to want the same things: fidget rings, slime, a Labubu key chain, a Squishmallow, a Sephora gift card, a digital wad of Robux, a hoverboard, and maybe a puppy. The adult who strives for a more bespoke level of gift-giving--or simply to find something with no connection to screens, mirrors, or fads--risks coming off as presumptuous and pretentious.

A.3 Proof of Theorem 1 In this subsection, we prove Theorem 1 with Assumption 1. Assumption 1. l = 1,, N, t = 1,, T, diag null As described in Sections 4.1 and 4.2, for gradients of OTTT, we have Remark 2. The above conclusion mainly focuses on the gradients for connection weights Remark 3. Note that the gradients based on spike representation may also include small errors since A.4 Proof of Theorem 2 In this subsection, we prove Theorem 2. Theorem 2. If Assumption 1 holds, As described in Sections 4.1 and 4.2 and similar to the proof of Theorem 1, let Remark 4. The above conclusion considers the single-layer condition. It can be generalized to the multi-layer condition. Therefore, the conclusion can be directly generalized to these conditions as well. L} based on the gradient-based optimizer. For VGG network structures, we directly impose sWS on all weights. For more illustrations and other details, please directly refer to [4].

Consequently, evaluating models on test splits that might have leaked into the training set is prone to misleading conclusions.

Specifically, we detail how baseline models are tuned in link prediction tasks as follows.