What is Implicit Deep Learning?

See a larger version of the cover image here. Prediction rules in deep learning are based on a forward, recursive computation through several layers. Implicit deep learning rules go much beyond, by relying on the solution of an implicit (or, "fixed-point") equation that has to be numerically solved in order to make the prediction: for a given input vector u, the predicted vector y is of the form Here, the so-called "state" n-vector x, which contains the hidden features of the model, is not expressed explicitly; rather it is implicitly defined via the "fixed-point" (or, equilibrium) equation x ϕ(Ax Bu). At first glance, the above models seem very specific. Perhaps surprisingly, they include a special case most known neural network architectures, including standard feedforward networks, CNNs, RNNs, and many more.