A wide range of empirical and theoretical works have shown that overparameterisation can amplify the performance of neural networks. According to the lottery ticket hypothesis, overparameterised networks have an increased chance of containing a sub-network that is well-initialised to solve the task at hand. A more parsimonious approach, inspired by animal learning, consists in guiding the learner towards solving the task by curating the order of the examples, i.e. providing a curriculum. However, this learning strategy seems to be hardly beneficial in deep learning applications. In this work, we undertake an analytical study that connects curriculum learning and overparameterisation. In particular, we investigate their interplay in the online learning setting for a 2-layer network in the XOR-like Gaussian Mixture problem. Our results show that a high degree of overparameterisation -- while simplifying the problem -- can limit the benefit from curricula, providing a theoretical account of the ineffectiveness of curricula in deep learning.

The left and right singular vectors are trained with all task losses, and the diagonal matrices are trained using taskspecific Compact neural network offers many benefits for losses. Our design is mainly inspired by analytical real-world applications. However, it is usually studies on overparameterised networks for MTL [Lampinen challenging to train the compact neural networks and Ganguli, 2018] that the training/test error dynamics depends with small parameter sizes and low computational on the time-evolving alignment of the network parameters costs to achieve the same or better model performance to the singular vectors of the training data, and a quantifiable compared to more complex and powerful task alignment describing the transfer benefits among architecture. This is particularly true for multitask multiple tasks depends on the singular values and input feature learning, with different tasks competing for resources.