RcTorch: a PyTorch Reservoir Computing Package with Automated Hyper-Parameter Optimization

Reservoir computers (RCs), also known as echo state networks, are specialized, artificial neural networks. They are powerful and train very fast. However, today RC is not as commonly employed by the machine learning community at large as other classes of neural network models. Unlike most neural networks, the majority of RC weights are not optimized via the backpropagation algorithm. Instead, the weights are generated via a stochastic process that is very sensitive to a few numbers, typically less than 10, called hyper-parameters (HPs). In a simple feed forward neural network, rather than being optimized during training, HPs govern the learning process. The optimization of HPs is a significant challenge to the widespread adoption of the RC. Other classes of neural networks, which are extremely popular today, have faced similar significant challenges in the past. At times, pessimism about neural networks and, thus, AI more broadly, has led to aversion to these models by the scientific community at large, resulting in an AI winter: a "period following a massive wave of hype for AI characterized by a disillusionment that causes a freeze in funding and publications"