Continuous Ant-Based Neural Topology Search

Manually optimizing artificial neural network (ANN) structures has been an obstacle to the advancement of machine learning as it is significantly time-consuming and requires a considerable level of domain expertise [1]. The structure of an ANN is typically chosen based on its reputation based on results of existent literature or based on knowledge shared across the machine learning community, however changing even a few problem-specific meta-parameters can lead to poor generalization upon committing to a specific topology [2, 3]. To address these challenges, a number of neural architecture search (NAS) [1, 4-8] and neuroevolution (NE) [9, 10] algorithms have been developed to automate the process of ANN design. More recently, nature-inspired neural architecture search (NINAS) algorithms have shown increasing promise, including the Artificial Bee Colony (ABC) optimization procedure [11], the Bat algorithm [12], the Firefly algorithm [13], and the Cuckoo Search algorithm [14]. Among the more recently successful applied NINAS strategies are those based on ant colony optimization (ACO) [15], which have proven to be particularly powerful when automating the design of recurrent neural networks (RNNs). Originally, ACO for NAS was limited to small structures based on Jordan and Elman RNNs [16] or was used as a process for reducing the number of network inputs [17]. Later work proposed generalizations of ACO for optimizing the synaptic connections of RNN memory cell structures [18] and even entire RNN architectures in an algorithmic framework called Ant-based Neural Topology Search (ANTS) [19]. In the ANTS process, ants traverse a single massively-connected "superstructure", which contains all of the possible ways that the nodes of an RNN may connect with each other, both in terms of structure (i.e., all possible feed forward connections), and in time (i.e., all possible recurrent synapses that span many different time delays), searching for optimal RNN sub-networks.