The 2024 Nobel Prize in Physics was awarded to John Hopfield and Geoffrey Hinton, "for foundational discoveries and inventions that enable machine learning with artificial neural networks." As noted by the Nobel committee, their work moved the boundaries of physics. This is a brief reflection on Hopfield's work, its implications for the emergence of biological physics as a part of physics, the path from his early papers to the modern revolution in artificial intelligence, and prospects for the future.

Princeton Center for Theoretical Physics, Princeton University, Princeton, New Jersey 08544 USA (Dated: December 13, 2021) We consider words as a network of interacting letters, and approximate the probability distribution of states taken on by this network. Despite the intuition that the rules of English spelling are highly combinatorial (and arbitrary), we find that maximum entropy models consistent with pairwise correlations among letters provide a surprisingly good approximation to the full statistics of four letter words, capturing 92% of the multi-information among letters and even'discovering' real words that were not represented in the data from which the pairwise correlations were estimated. The maximum entropy model defines an energy landscape on the space of possible words, and local minima in this landscape account for nearly two-thirds of words used in written English. Many complex systems convey an impression of order into these controversies about language in the broad that is not so easily captured by the traditional tools of sense, but rather to test the power of pairwise interactions theoretical physics. Thus, it is not clear what sort of to capture seemingly complex structure.

We argue that K-means and deterministic annealing algorithms for geometric clustering can be derived from the more general Information Bottleneck approach. If we cluster the identities of data points to preserve information about their location, the set of optimal solutions is massively degenerate. But if we treat the equations that define the optimal solution as an iterative algorithm, then a set of "smooth" initial conditions selects solutions with the desired geometrical properties. In addition to conceptual unification, we argue that this approach can be more efficient and robust than classic algorithms.

What happens to the optimal interpretation of noisy data when there exists more than one equally plausible interpretation of the data? In a Bayesian model-learning framework the answer depends on the prior expectations of the dynamics of the model parameter that is to be inferred from the data. Local time constraints on the priors are insufficient to pick one interpretation over another. On the other hand, nonlocal time constraints, induced by a 1/f noise spectrum of the priors, is shown to permit learning of a specific model parameter even when there are infinitely many equally plausible interpretations of the data. This transition is inferred by a remarkable mapping of the model estimation problem to a dissipative physical system, allowing the use of powerful statistical mechanical methods to uncover the transition from indeterminate to determinate model learning.

In this paper we introduce another prior over the dynamics of a(t). We assume that fluctuations in a(t) have a 1 / f spectrum, as observed ubiquitously in nature.

We introduce an information theoretic method for nonparametric, nonlinear dimensionalityreduction, based on the infinite cluster limit of rate distortion theory. By constraining the information available to manifold coordinates, a natural probabilistic map emerges that assigns original data to corresponding points on a lower dimensional manifold. With only the information-distortion trade off as a parameter, our method determines theshape of the manifold, its dimensionality, the probabilistic map and the prior that provide optimal description of the data.

We argue that K-means and deterministic annealing algorithms for geometric clusteringcan be derived from the more general Information Bottleneck approach.If we cluster the identities of data points to preserve information about their location, the set of optimal solutions is massively degenerate. But if we treat the equations that define the optimal solution as an iterative algorithm, then a set of "smooth" initial conditions selects solutions with the desired geometrical properties. In addition to conceptual unification,we argue that this approach can be more efficient and robust than classic algorithms.

From olfaction to vision and audition, there is an increasing need, and a growing number of experiments [1]-[8] that study responses of sensory neurons to natural stimuli. Natural stimuli have specific statistical properties [9, 10], and therefore sample only a subspace of all possible spatial and temporal frequencies explored during stimulation with white noise. Observing the full dynamic range of neural responses may require using stimulus ensembles which approximate those occurring in nature, and it is an attractive hypothesis that the neural representation of these natural signals may be optimized in some way. Finally, some neuron responses are strongly nonlinear and adaptive, and may not be predicted from a combination of responses to simple stimuli. It has also been shown that the variability in neural response decreases substantially when dynamical, rather than static, stimuli are used [11, 12]. For all these reasons, it would be attractive to have a rigorous method of analyzing neural responses to complex, naturalistic inputs.

A population of neurons typically exhibits a broad diversity of responses to sensory inputs. The intuitive notion of functional classification is that cells can be clustered so that most of the diversity is captured by the identity of the clusters rather than by individuals within clusters. We show how this intuition can be made precise using information theory, without any need to introduce a metric on the space of stimuli or responses. Applied to the retinal ganglion cells of the salamander, this approach recovers classical results, but also provides clear evidence for subclasses beyond those identified previously. Further, we find that each of the ganglion cells is functionally unique, and that even within the same subclass only a few spikes are needed to reliably distinguish between cells.

From olfaction to vision and audition, there is an increasing need, and a growing number of experiments [1]-[8] that study responses of sensory neurons to natural stimuli. Natural stimuli have specific statistical properties [9, 10], and therefore sample only a subspace of all possible spatial and temporal frequencies explored during stimulation with white noise. Observing the full dynamic range of neural responses may require using stimulus ensembles whichapproximate those occurring in nature, and it is an attractive hypothesis that the neural representation of these natural signals may be optimized in some way. Finally, some neuron responses are strongly nonlinear and adaptive, and may not be predicted from a combination of responses to simple stimuli. It has also been shown that the variability in neural response decreases substantially when dynamical, rather than static, stimuli are used [11, 12]. For all these reasons, it would be attractive to have a rigorous method of analyzing neural responses to complex, naturalistic inputs.