Due to the high computational demands executing a rigorous comparison between hyperparameter optimization (HPO) methods is often cumbersome. The goal of this paper is to facilitate a better empirical evaluation of HPO methods by providing benchmarks that are cheap to evaluate, but still represent realistic use cases. We believe these benchmarks provide an easy and efficient way to conduct reproducible experiments for neural hyperparameter search. Our benchmarks consist of a large grid of configurations of a feed forward neural network on four different regression datasets including architectural hyperparameters and hyperparameters concerning the training pipeline. Based on this data, we performed an in-depth analysis to gain a better understanding of the properties of the optimization problem, as well as of the importance of different types of hyperparameters. Second, we exhaustively compared various different state-of-the-art methods from the hyperparameter optimization literature on these benchmarks in terms of performance and robustness.
I think one thing that might help... look at it like this. A standard densely connected layer can represent a convolutional layer (standard'under the hood' implementation of convolution layers even converts it into a dense layer in a lot of cases so it can leverage fast matrix operations). In theory, if the dense layer can represent a convolutional layer, why's the convolutional layer used instead? You could just say that it's because there's less parameters, but it goes deeper than that. It'makes an assumption' that things likely to be seen in the dataset should be translation equivarient.
In the wonderful world of machine learning and artificial intelligence, there exists this structure called an autoencoder. Autoencoders are a type neural network which is part of unsupervised learning (or, to some, semi-unsupervised learning). There are many different types of autoencoders used for many purposes, some generative, some predictive, etc. This article should provide you with a toolbox and guide to the different types of autoencoders. The basic type of an autoencoder looks like the one above.
In machine learning, a deep belief network (DBN) is a generative graphical model, or alternatively a type of deep neural network, composed of multiple layers of latent variables ("hidden units"), with connections between the layers but not between units within each layer. When trained on a set of examples in an unsupervised way, a DBN can learn to probabilistically reconstruct its inputs. The layers then act as feature detectors on inputs. After this learning step, a DBN can be further trained in a supervised way to perform classification. DBNs can be viewed as a composition of simple, unsupervised networks such as restricted Boltzmann machines (RBMs) or autoencoders, where each sub-network's hidden layer serves as the visible layer for the next.