Not all generate-and-test search algorithms are created equal. Bayesian Optimization (BO) invests a lot of computation time to generate the candidate solution that best balances the predicted value and the uncertainty given all previous data, taking increasingly more time as the number of evaluations performed grows. Evolutionary Algorithms (EA) on the other hand rely on search heuristics that typically do not depend on all previous data and can be done in constant time. Both the BO and EA community typically assess their performance as a function of the number of evaluations. However, this is unfair once we start to compare the efficiency of these classes of algorithms, as the overhead times to generate candidate solutions are significantly different. We suggest to measure the efficiency of generate-and-test search algorithms as the expected gain in the objective value per unit of computation time spent. We observe that the preference of an algorithm to be used can change after a number of function evaluations. We therefore propose a new algorithm, a combination of Bayesian optimization and an Evolutionary Algorithm, BEA for short, that starts with BO, then transfers knowledge to an EA, and subsequently runs the EA. We compare the BEA with BO and the EA. The results show that BEA outperforms both BO and the EA in terms of time efficiency, and ultimately leads to better performance on well-known benchmark objective functions with many local optima. Moreover, we test the three algorithms on nine test cases of robot learning problems and here again we find that BEA outperforms the other algorithms.

In this paper, the fuzzy multi-objective reliability redundancy allocation problem (FMORRAP) is proposed, which maximizes the system reliability while simultaneously minimizing the system cost under the type 2 fuzzy uncertainty. In the proposed formulation, the higher order uncertainties (such as parametric, manufacturing, environmental, and designers uncertainty) associated with the system are modeled with interval type 2 fuzzy sets (IT2 FS). The footprint of uncertainty of the interval type 2 membership functions (IT2 MFs) accommodates these uncertainties by capturing the multiple opinions from several system experts. We consider IT2 MFs to represent the subsystem reliability and cost, which are to be further aggregated using extension principle to evaluate the total system reliability and cost according to their configurations, i.e., series parallel and parallel series. We proposed a particle swarm optimization (PSO) based novel solution approach to solve the FMORRAP. To demonstrate the applicability of two formulations, namely, series parallel FMORRAP and parallel series FMORRAP, we performed experimental simulations on various numerical data sets. The decision makers/system experts assign different importance to the objectives (system reliability and cost), and these preferences are represented by sets of weights. The optimal results are obtained from our solution approach, and the Pareto optimal front is established using these different weight sets. The genetic algorithm (GA) was implemented to compare the results obtained from our proposed solution approach. A statistical analysis was conducted between PSO and GA, and it was found that the PSO based Pareto solution outperforms the GA.

The present and future of evolutionary algorithms depends on the proper use of modern parallel and distributed computing infrastructures. Although still sequential approaches dominate the landscape, available multi-core, many-core and distributed systems will make users and researchers to more frequently deploy parallel version of the algorithms. In such a scenario, new possibilities arise regarding the time saved when parallel evaluation of individuals are performed. And this time saving is particularly relevant in Genetic Programming. This paper studies how evaluation time influences not only time to solution in parallel/distributed systems, but may also affect size evolution of individuals in the population, and eventually will reduce the bloat phenomenon GP features. This paper considers time and space as two sides of a single coin when devising a more natural method for fighting bloat. This new perspective allows us to understand that new methods for bloat control can be derived, and the first of such a method is described and tested. Experimental data confirms the strength of the approach: using computing time as a measure of individuals' complexity allows to control the growth in size of genetic programming individuals.

Genetic Algorithms (GAs) are members of a general class of optimization algorithms, known as Evolutionary Algorithms (EAs), which simulate a fictional environment based on theory of evolution to deal with various types of mathematical problem, especially those related to optimization. Also Genetic Algorithms can be categorized as a subset of Metaheuristics, which are general-purpose tools and algorithms to solve optimization and unsupervised learning problems. In this series of video tutorials, we are going to learn about Genetic Algorithms, from theory to implementation. After having a brief review of theories behind EA and GA, two main versions of genetic algorithms, namely Binary Genetic Algorithm and Real-coded Genetic Algorithm, are implemented from scratch and line-by-line, using both Python and MATLAB. This course is instructed by Dr. Mostapha Kalami Heris, who has years of practical work and active teaching in the field of computational intelligence.

One of the key difficulties in using estimation-of-distribution algorithms is choosing the population sizes appropriately: Too small values lead to genetic drift, which can cause enormous difficulties. In the regime with no genetic drift, however, often the runtime is roughly proportional to the population size, which renders large population sizes inefficient. Based on a recent quantitative analysis which population sizes lead to genetic drift, we propose a parameter-less version of the compact genetic algorithm that automatically finds a suitable population size without spending too much time in situations unfavorable due to genetic drift. We prove an easy mathematical runtime guarantee for this algorithm and conduct an extensive experimental analysis on four classic benchmark problems. The former shows that under a natural assumption, our algorithm has a performance similar to the one obtainable from the best population size. The latter confirms that missing the right population size can be highly detrimental and shows that our algorithm as well as a previously proposed parameter-less one based on parallel runs avoids such pitfalls. Comparing the two approaches, ours profits from its ability to abort runs which are likely to be stuck in a genetic drift situation.

Automatic programming (AP) is an important area of Machine Learning (ML) where computer programs are generated automatically. Swarm Programming (SP), a newly emerging research area in AP, automatically generates the computer programs using Swarm Intelligence (SI) algorithms. This paper presents two grammar-based SP methods named as Grammatical Moth-Flame Optimizer (GMFO) and Grammatical Whale Optimizer (GWO). The Moth-Flame Optimizer and Whale Optimization algorithm are used as search engines or learning algorithms in GMFO and GWO respectively. The proposed methods are tested on Santa Fe Ant Trail, quartic symbolic regression, and 3-input multiplexer problems. The results are compared with Grammatical Bee Colony (GBC) and Grammatical Fireworks algorithm (GFWA). The experimental results demonstrate that the proposed SP methods can be used in automatic computer program generation.

Design of experiments, random search, initialization of population-based methods, or sampling inside an epoch of an evolutionary algorithm use a sample drawn according to some probability distribution for approximating the location of an optimum. Recent papers have shown that the optimal search distribution, used for the sampling, might be more peaked around the center of the distribution than the prior distribution modelling our uncertainty about the location of the optimum. We confirm this statement, provide explicit values for this reshaping of the search distribution depending on the population size $\lambda$ and the dimension $d$, and validate our results experimentally.

In symbolic regression, the search for analytic models is typically driven purely by the prediction error observed on the training data samples. However, when the data samples do not sufficiently cover the input space, the prediction error does not provide sufficient guidance toward desired models. Standard symbolic regression techniques then yield models that are partially incorrect, for instance, in terms of their steady-state characteristics or local behavior. If these properties were considered already during the search process, more accurate and relevant models could be produced. We propose a multi-objective symbolic regression approach that is driven by both the training data and the prior knowledge of the properties the desired model should manifest. The properties given in the form of formal constraints are internally represented by a set of discrete data samples on which candidate models are exactly checked. The proposed approach was experimentally evaluated on three test problems with results clearly demonstrating its capability to evolve realistic models that fit the training data well while complying with the prior knowledge of the desired model characteristics at the same time. It outperforms standard symbolic regression by several orders of magnitude in terms of the mean squared deviation from a reference model.

It is becoming evident each and every day that machine learning algorithms are achieving impressive results in domains in which it is hard to specify a set of rules for their procedures. Examples of this phenomenon include industries like finance [49, 5], transportation [37], education [42, 22], health care [23] and tasks like image recognition [41, 16, 17], machine translation [43, 7], and speech recognition [46, 24, 53, 50]. Motivated by the ease of adoption and the increased availability of affordable computational power (especially cloud computing services), machine learning algorithms are being explored in almost every commercial application and are offering great promise for the future of automation. Facing such a vast adoption across multiple disciplines, some of their weaknesses are exposed and sometimes exploited by malicious actors. For example, a common challenge to these algorithms is "generalization" or "robustness", which is the ability of the algorithm to maintain performance whenever dealing with data coming from a different distribution with which it was trained. For a long period of time, the sole focus of machine learning researchers was improving the performance of machine learning systems (true positive rate, accuracy, etc.). Nowadays, the robustness of these systems can no longer be ignored; many of them have been shown to be highly vulnerable to intentional adversarial attacks.

A major goal of machine learning is to create techniques that abstract away irrelevant information. The generalisation property of standard Learning Classifier System (LCS) removes such information at the feature level but not at the feature interaction level. Code Fragments (CFs), a form of tree-based programs, introduced feature manipulation to discover important interactions, but they often contain irrelevant information, which causes structural inefficiency. XOF is a recently introduced LCS that uses CFs to encode building blocks of knowledge about feature interaction. This paper aims to optimise the structural efficiency of CFs in XOF. We propose two measures to improve constructing CFs to achieve this goal. Firstly, a new CF-fitness update estimates the applicability of CFs that also considers the structural complexity. The second measure we can use is a niche-based method of generating CFs. These approaches were tested on Even-parity and Hierarchical problems, which require highly complex combinations of input features to capture the data patterns. The results show that the proposed methods significantly increase the structural efficiency of CFs, which is estimated by the rule "generality rate". This results in faster learning performance in the Hierarchical Majority-on problem. Furthermore, a user-set depth limit for CF generation is not needed as the learning agent will not adopt higher-level CFs once optimal CFs are constructed.