PReLU: Yet Another Single-Layer Solution to the XOR Problem

The XOR problem has traditionally been used to illustrate the limitations of single-layer networks since Minsky and Papert's seminal work [8], which even contributed to the first AI Winter [12]. It has traditionally required at least one hidden layer to solve, making it a litmus test for network complexity. Trivially, any function, no matter how complex, can be learned in a single layer by just using itself as the activation function, and that says nothing about its general applicability and usefulness. Here, however, we reveal this ability in a simple, general and well-established activation function. This study demonstrates how using the Parametric Rectified Linear Unit (PReLU) activation [4] overcomes these limitations, effectively solving the XOR problem without additional layers. This ability has significant implications for neural network design and efficiency, potentially leading to simpler architectures for complex problems. On another front, recent advancements in neuroscience have revealed that individual human neocortical pyramidal neurons can learn to compute the XOR function [3]. This discovery has inspired new artificial neuron models and activation functions that aim to bridge the gap between biological and artificial neurons [9]. Albeit not producing the same activation curves as the ones found in biological neurons, the PReLU activation matches their representational power, at least regarding the XOR function.