A Experimental Setup

We provide detailed experimental setup in this section. We empirically show that even a simple linear network can enter the EOS regime. Fully-connected Network We conduct further experiments on several different fully-connected networks with 4 hidden layers with various activation functions. For example, the structure of a fully-connected tanh network is shown in Table 2. Table 2: Fully-connected network Layer # Name Layer In shape Out shape 1 Flatten() (3, 32, 32) 3072 2 fc1 nn.Linear(3072,200,bias=False) 3072 200 3 nn.tanh() Like the fully-connected network experiments, we consider tanh, ReLU and ELU activations.