Why are neural networks so powerful?

It is common knowledge that neural networks are very powerful and they can be used for almost any statistical learning problem with great results. But have you thought about why is this the case? Why is this method more powerful in most scenarios than many other algorithms? As always with machine learning, there is a precise mathematical reason for this. Simply saying, the set of functions described by a neural network model is very large.