Encouraging an Appropriate Representation Simplifies Training of Neural Networks

A common assumption about neural networks is that they can learn an appropriate internal representations on their own, see e.g. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalisation ability of neural networks.