On the expressivity of deep Heaviside networks

The Heaviside activation function is for instance used in Hopfield networks [ 1 ] that have recently seen a resurge due to their connections t o attention layers [ 2, 3 ] and the 2024 Nobel Prize in Physics that was partially award ed for their development. Moreover, the Heaviside activation function is closely related to quantized neural networks [ 4, 5 ], playing a key role in enabling energy efficient deployment o f large language models (LLMs) [ 6, 7 ]. We refer to neural networks with several hidden layers and th e Heaviside activation function as deep Heaviside (neural) networks (DHNs). These networks are also known as (linear) threshold networks. The Heaviside activation function can be traced back to the fi rst attempts to build an artificial counterpart of a biological neuron. In the brain, the inputs of a neuron contribute to its membrane potential and the neuron discharges/fires if th e membrane potential exceeds a certain threshold.