A Fully Connected Disconnect

One of the fundamental building blocks in deep learning is the use of one or more fully connected dense "hidden" layers. This typically takes the form of the primary layer type throughout the network or as the final layer(s) in the more novel neural network architectures. This fully connected layer consists of all the input elements connecting to all of the processing units (aka neurons) in the hidden layer. All of the input elements are then processed by all of the processing units in that layer. My observation of this topology is that although this method is clearly effective it doesn't appear to be very efficient.