Variational Garrote for Statistical Physics-based Sparse and Robust Variable Selection

Identifying relationships between variables is a fundamental task in science. Among various approaches, linear regression plays a central role in linking explanatory variables to dependent variables in statistical modeling [1, 2]. Linear regression is useful in physics [3, 4] for extracting equations of motion from time series data [5] and for predicting trends in dynamical systems [6], but its simplicity, interpretability, and predictive power make it a cornerstone of data analysis [7], forecasting [8], and decision-making [9] in many fields. Moreover, linear regression forms the foundation for many advanced statistical and machine learning models [10], including logistic regression [11], support vector machines [12], and generalized linear models [13]. Extensions of linear regression often aim to capture more complex relationships by introducing higher-order polynomial terms or additional nonlinear transformations. Modern developments in machine learning have enabled the training of deep and highly overparameterized models capable of modeling intricate patterns far beyond the reach of simple linear approaches. In particular, deep learning models can be interpreted as sophisticated forms of nonlinear regression [14], capable of approximating complex functions with high flexibility. Despite its utility, linear regression struggles with modern high-dimensional datasets where only a small subset of variables is truly informative.