Visualizing Neural Networks

So the question I have is: what does the frontier of the space of optimal networks look like, what are the inherent limits of depth vs expressivity of these models, and are there dimensional scaling laws that can describe all this in an information theoretic way? This recent paper gives a great treatment on the expressivity of convolution networks by using a deep layered architecture that generalizes convolutional neural networks called sim-nets. As a simple first step I wanted to see what could be done to visualize the operations a deep neural net performs. So I constructed a standard network that takes vector inputs of size 2 and produces vector outputs of size 3 which we can think of as a mapping of the cartesian plane into RGB color space. Taking many copies of this net and randomly initializing them, (with normally distributed weights and biases) we can plot them in a grid and see the networks' outputs as a set of images.