This paper demonstrates that grokking behavior in modular arithmetic with a modulus P in a neural network can be controlled by modifying the profile of the activation function as well as the depth and width of the model. Plotting the even PCA projections of the weights of the last NN layer against their odd projections further yields patterns which become significantly more uniform when the nonlinearity is increased by incrementing the number of layers. These patterns can be employed to factor P when P is nonprime. Finally, a metric for the generalization ability of the network is inferred from the entropy of the layer weights while the degree of nonlinearity is related to correlations between the local entropy of the weights of the neurons in the final layer.

While neural networks (NN) were first proposed in 1943 [1], initial implementations were restricted to networks with a small number of neurons and one or two layers[2], [3]. This limitation was eliminated through the backpropagation training algorithm[3], [4], [5] in conjunction with exponential improvements in computational performance. The resulting procedure generates a system model exclusively from experimental or simulated data and can accordingly be employed in a wide variety of scientific and engineering fields. In particular, a system, which typically can be characterized by a few coordinates and equations, is instead described by a large number of variables that interact nonlinearly. By optimizing a loss function, which may be further subject to physical constraints as in physics-informed machine 1 learning,[6] the parameters associated with the interactions are adjusted to approximate the data. The trained model then can predict the response of the system to unobserved input data. Although such an approach possesses significant advantages in terms of generality and simplicity, it lacks the precision and efficiency afforded by the solution of deterministic equations. Similarly, the large dimensionality of the representation obscures the underlying physics and mathematics. For complex systems, however, especially in the presence of stochastic noise or measurement inaccuracy, procedures based on numerical optimization can be effectively optimal.[7],

This paper introduces a modified variational autoencoder (VAEs) that contains an additional neural network branch. The resulting branched VAE (BVAE) contributes a classification component based on the class labels to the total loss and therefore imparts categorical information to the latent representation. As a result, the latent space distributions of the input classes are separated and ordered, thereby enhancing the classification accuracy. The degree of improvement is quantified by numerical calculations employing the benchmark MNIST dataset for both unrotated and rotated digits. The proposed technique is then compared to and then incorporated into a VAE with fixed output distributions. This procedure is found to yield improved performance for a wide range of output distributions.

This paper examines the relationship between the behavior of a neural network and the distribution formed from the projections of the data records into the space spanned by the low-order principal components of the training data. For example, in a benchmark calculation involving rotated and unrotated MNIST digits, classes (digits) that are mapped far from the origin in a low-dimensional principal component space and that overlap minimally with other digits converge rapidly and exhibit high degrees of accuracy in neural network calculations that employ the associated components of each data record as inputs. Further, if the space spanned by these low-order principal components is divided into bins and the input data records that are mapped into a given bin averaged, the resulting pattern can be distinguished by its geometric features which interpolate between those of adjacent bins in an analogous manner to variational autoencoders. Based on this observation, a simply realized data balancing procedure can be realized by evaluating the entropy associated with each histogram bin and subsequently repeating the original image data associated with the bin by a number of times that is determined from this entropy.