These authors contributed equally to this work.

Backdoor learning is an emerging and vital topic for studying deep neural networks'

Recurrent neural networks have a strong inductive bias towards learning temporally compressed representations, as the entire history of a sequence is represented by a single vector. By contrast, Transformers have little inductive bias towards learning temporally compressed representations, as they allow for attention over all previously computed elements in a sequence. Having a more compressed representation of a sequence may be beneficial for generalization, as a high-level representation may be more easily re-used and re-purposed and will contain fewer irrelevant details. At the same time, excessive compression of representations comes at the cost of expressiveness. We propose a solution which divides computation into two streams. A slow stream that is recurrent in nature aims to learn a specialized and compressed representation, by forcing chunks of K time steps into a single representation which is divided into multiple vectors. At the same time, a fast stream is parameterized as a Transformer to process chunks consisting of K time-steps conditioned on the information in the slow-stream. In the proposed approach we hope to gain the expressiveness of the Transformer, while encouraging better compression and structuring of representations in the slow stream. We show the benefits of the proposed method in terms of improved sample efficiency and generalization performance as compared to various competitive baselines for visual perception and sequential decision making tasks.

As we explain next, this is made possible by sparse modeling with CSC-layers. 2 B.1 Method We apply our visualization method described above to the SDNet-18 trained on ImageNet (see Sec. 4) The results are provided in Figure B.1. It can be observed that the shallow layers (e.g., layer 1 - 5) capture rich details of the SDNet-18 progressively remove some of the unrelated details from the network input. In Figure C.1, we provide a visualization of the learned dictionary in the first layer of SDNet-18 The result shows our CSC-layer feature maps are highly sparse. Table D.1 shows the comparison of SDNet-18/34 and SDNet-18/34-All on CIFAR-10, CIFAR-100 Both models have high accuracy performance while SDNet-18 is significantly faster. Figure B.1: Visualization of feature maps for 5 images at selected layers of a SDNet-18 trained on Figure C.1: Visualization of the learned dictionary of first layer of SDNet-18-All trained on ImageNet.

Despite strong empirical performance for image classification, deep neural networks are often regarded as "black boxes" and they are difficult to interpret.

Implicit neural networks, a.k.a., deep equilibrium networks, are a class of implicit-depth learning models where function evaluation is performed by solving a fixed

Deep equilibrium models (DEQs) refrain from the traditional layer-stacking paradigm and turn to find the fixed point of a single layer.