This paper focuses on a class of reinforcement learning problems where significant events are rare and limited to a single positive reward per episode. A typical example is that of an agent who has to choose a partner to cooperate with, while a large number of partners are simply not interested in cooperating, regardless of what the agent has to offer. We address this problem in a continuous state and action space with two different kinds of search methods: a gradient policy search method and a direct policy search method using an evolution strategy. We show that when significant events are rare, gradient information is also scarce, making it difficult for policy gradient search methods to find an optimal policy, with or without a deep neural architecture. On the other hand, we show that direct policy search methods are invariant to the rarity of significant events, which is yet another confirmation of the unique role evolutionary algorithms has to play as a reinforcement learning method.

This paper proposes a model-based framework to automatically and efficiently design understandable and verifiable behaviors for swarms of robots. The framework is based on the automatic extraction of two distinct models: 1) a neural network model trained to estimate the relationship between the robots' sensor readings and the global performance of the swarm, and 2) a probabilistic state transition model that explicitly models the local state transitions (i.e., transitions in observations from the perspective of a single robot in the swarm) given a policy. The models can be trained from a data set of simulated runs featuring random policies. The first model is used to automatically extract a set of local states that are expected to maximize the global performance. These local states are referred to as desired local states. The second model is used to optimize a stochastic policy so as to increase the probability that the robots in the swarm observe one of the desired local states. Following these steps, the framework proposed in this paper can efficiently lead to effective controllers. This is tested on four case studies, featuring aggregation and foraging tasks. Importantly, thanks to the models, the framework allows us to understand and inspect a swarm's behavior. To this end, we propose verification checks to identify some potential issues that may prevent the swarm from achieving the desired global objective. In addition, we explore how the framework can be used in combination with a "standard" evolutionary robotics strategy (i.e., where performance is measured via simulation), or with online learning.

Robots deployed at orders of magnitude different size scales, and that retain the same desired behavior at any of those scales, would greatly expand the environments in which the robots could operate. However it is currently not known whether such robots exist, and, if they do, how to design them. Since self similar structures in nature often exhibit self similar behavior at different scales, we hypothesize that there may exist robot designs that have the same property. Here we demonstrate that this is indeed the case for some, but not all, modular soft robots: there are robot designs that exhibit a desired behavior at a small size scale, and if copies of that robot are attached together to realize the same design at higher scales, those larger robots exhibit similar behavior. We show how to find such designs in simulation using an evolutionary algorithm. Further, when fractal attachment is not assumed and attachment geometries must thus be evolved along with the design of the base robot unit, scale invariant behavior is not achieved, demonstrating that structural self similarity, when combined with appropriate designs, is a useful path to realizing scale invariant robot behavior. We validate our findings by demonstrating successful transferal of self similar structure and behavior to pneumatically-controlled soft robots. Finally, we show that biobots can spontaneously exhibit self similar attachment geometries, thereby suggesting that self similar behavior via self similar structure may be realizable across a wide range of robot platforms in future.

Bio-inspired computing focuses on extracting computational models for problem solving from in-depth understanding of behaviour and mechanisms of biological systems. In recent years, cellular computational models based on the structure and the processes of living cells, such as bacterial colonies [43] and viral models [23] have become an important line of research in bio-inspired computing. Physarum-computing, as an example of cellular computing model, has attracted the attention of many researchers [84]. Physarum polycephalum (Physarum for short) is an example of plasmodial slime moulds that are classified as a fungus "Myxomycetes" [21]. In recent years, research on Physarum-inspired computing has become more popular since Nakagaki et al. (2000) performed their well-known experiments showing that Physarum was able to find the shortest route through a maze [57]. Recent research has confirmed the ability of Physarum-inspired algorithms to solve a wide range of problems [103, 78]. Physarum can be modelled as a reaction-diffusion system (cytoplasmic liquid) encapsulated in an elastic growing membrane of actin-myosin cytoskeleton [2].

Valiant (2009) introduced a framework for a quantitative approach to evolution, called evolvability. The idea is, roughly, that there is an ideal behavior in every environment and the feedback that the various organisms receive during evolution indicates how close their behavior is to ideal. Ultimately, evolvability aims at modeling and explaining mechanisms that allow near-optimal behavior of organisms while exploiting realistic computational resources. Due to a result by Feldman (2008), evolvability is equivalent to learning in the correlational statistical query (CSQ) model (Bshouty & Feldman, 2002). Thus, evolvability algorithms correspond to a special type of local search learning algorithms that fall under the umbrella of the probably approximately correct (PAC) model of learning (Valiant, 1984).

The no free lunch theorem states that no model is better suited to every problem. A question that arises from this is how to design methods that propose optimizers tailored to specific problems achieving state-of-the-art performance. This paper addresses this issue by proposing the use of meta-learning to infer population-based black-box optimizers that can automatically adapt to specific classes of problems. We suggest a general modeling of population-based algorithms that result in Learning-to-Optimize POMDP (LTO-POMDP), a meta-learning framework based on a specific partially observable Markov decision process (POMDP). From that framework's formulation, we propose to parameterize the algorithm using deep recurrent neural networks and use a meta-loss function based on stochastic algorithms' performance to train efficient data-driven optimizers over several related optimization tasks. The learned optimizers' performance based on this implementation is assessed on various black-box optimization tasks and hyperparameter tuning of machine learning models. Our results revealed that the meta-loss function encourages a learned algorithm to alter its search behavior so that it can easily fit into a new context. Thus, it allows better generalization and higher sample efficiency than state-of-the-art generic optimization algorithms, such as the Covariance matrix adaptation evolution strategy (CMA-ES).

Sperm cells are genetically haploid, but because of the cytoplasmic bridges that link cells, they can be transcriptionally diploid. However, some gene transcripts are not shared. Bhutani et al. sequenced single sperm from mice, cattle, and macaques to determine the extent of distortion in the expression of these putatively selfish transcripts, which the authors call genoinformative markers (GIMs). Investigating the evolutionary pressures on these GIMs, they found that they exhibited signatures of positive selection yet tend to be biased toward sperm function. This observation explains why, relative to other tissues, testis shows distinctive gene expression patterns. Science , this issue p. [eabb1723][1] ### INTRODUCTION Mendel’s first law dictates that alleles are distributed equally among progeny, which requires sperm to be functionally independent of their genetic payload. Although mammalian spermatogenesis includes a long haploid stage with extensive gene expression, gene products are shared through spermatid cytoplasmic bridges, which decreases phenotypic differences between individual haploid sperm. However, there are known exceptions to the rule of complete sharing of transcripts between spermatids, including a gene product encoded by the t haplotype in mice that has restricted localization and therefore does not cross cytoplasmic bridges. This results in functional differences between sperm depending on their genotype and leads to preferential transmission of the t haplotype, causing it to act as a selfish genetic element. ### RATIONALE Given that many mammalian transcripts have specific subcellular localization, we reasoned that many spermatid gene products would not be completely shared across cytoplasmic bridges and would therefore exhibit allele-specific bias reflecting the haploid genotype of the cell. Here, we performed single-cell RNA sequencing in four mammalian species to quantify allele-specific biases in spermatids and develop a new computational technique to jointly infer genotype and allelic expression biases (both technical and biological) in single haploid cells. ### RESULTS We show that a large class of mammalian genes exhibit allelic bias linked to the haploid genotype of the cell, which we call “genoinformative markers” (GIMs). Confident GIMs comprise 31 to 52% of spermatid-expressed genes in mice, bulls, cynomolgus macaques, and humans, and at least 62% of GIMs had at least a twofold mean allelic bias. Most GIMs identified were autosomal genes, but we did find evidence for genoinformativity in sex chromosome genes as well. Genoinformativity tends to be conserved between individuals and between homologs of different species, suggesting that it is governed by slow-evolving features. GIMs are enriched for specific subcellular localization patterns and 3ʹ untranslated region motifs and are depleted from chromatoid bodies (subcellular structures that transit across cytoplasmic bridges), consistent with the model of genoinformativity by evasion of cytoplasmic bridges. For GIMs with proteins that are also not shared across cytoplasmic bridges, sperm function may be affected by allelic differences, leading to sperm competition and sperm-level natural selection. Genes expressed in spermatogenesis are known to experience heightened selective forces on average, but a subset of GIMs experience further increases, as evidenced by statistically significant enrichment for signatures of selective sweeps, loss-of-function intolerance, and transmission ratio distortion in humans, mice, and bulls. These forces are consistent with a subset of GIMs acting as selfish genetic elements that spread alleles unevenly. For GIMs with functions both in sperm and in somatic tissues, this could cause an evolutionary conflict for genes because optimal function in highly specialized sperm cells may be detrimental in somatic cells. We identified evolutionary pressure to avoid this conflict, because GIMs are significantly enriched for testis-specific gene expression, paralogs, and isoforms. ### CONCLUSION Our work demonstrates that at least one-third of spermatid transcripts are GIMs across a variety of mammalian species. Many of the corresponding proteins may not have genoinformative expression, but we provide evolutionary evidence that a subset of GIMs show signatures of sperm-level natural selection, implying an effect of protein-level genoinformativity during mammalian evolution. This phenomenon may help to explain why testis gene expression patterns are an outlier relative to all other tissues, with a large fraction of testis-specific gene expression, paralogs, and isoforms. ![][2] Incomplete transcript sharing leads to sperm-level functional differences. Left: Models for sharing of gene products between adjacent haploid spermatids across cytoplasmic bridges. Genes with complete sharing (top) have no differences in allelic expression; genes with no sharing (middle) lead to allele-specific gene expression and potentially phenotypic differences in mature sperm; genes with partial sharing (bottom) lead to quantitative allelic biases and potential quantitative phenotypic differences in mature sperm. Most GIMs fall into the partial sharing category. Right: Phenotypic differences between mature sperm that are linked to the genotype (top) can lead to sperm-level natural selection, which may conflict with organism-level natural selection. This creates an evolutionary pressure that may be resolved by testis-specific expression, paralogs, and alternative isoforms. This suggests a possible explanation for the high prevalence of testis-specific gene expression patterns in mammals. Sperm are haploid but must be functionally equivalent to distribute alleles equally among progeny. Accordingly, gene products are shared through spermatid cytoplasmic bridges that erase phenotypic differences between individual haploid sperm. Here, we show that a large class of mammalian genes are not completely shared across these bridges. We call these genes “genoinformative markers” (GIMs) and show that a subset can act as selfish genetic elements that spread alleles unevenly through murine, bovine, and human populations. We identify evolutionary pressure to avoid conflict between sperm and somatic function as GIMs are enriched for testis-specific gene expression, paralogs, and isoforms. Therefore, GIMs and sperm-level natural selection may help to explain why testis gene expression patterns are an outlier relative to all other tissues. [1]: /lookup/doi/10.1126/science.abb1723 [2]: pending:yes

The genetic algorithm is a popular evolutionary algorithm. It uses Darwin's theory of natural evolution to solve complex problems in computer science. But, to do so, the algorithm's parameters need a bit of adjusting. One of the key parameters is mutation. It makes random changes in the chromosomes (i.e.

Modeling real-world phenomena is a focus of many science and engineering efforts, such as ecological modeling and financial forecasting, to name a few. Building an accurate model for complex and dynamic systems improves understanding of underlying processes and leads to resource efficiency. Towards this goal, knowledge-driven modeling builds a model based on human expertise, yet is often suboptimal. At the opposite extreme, data-driven modeling learns a model directly from data, requiring extensive data and potentially generating overfitting. We focus on an intermediate approach, model revision, in which prior knowledge and data are combined to achieve the best of both worlds. In this paper, we propose a genetic model revision framework based on tree-adjoining grammar (TAG) guided genetic programming (GP), using the TAG formalism and GP operators in an effective mechanism to incorporate prior knowledge and make data-driven revisions in a way that complies with prior knowledge. Our framework is designed to address the high computational cost of evolutionary modeling of complex systems. Via a case study on the challenging problem of river water quality modeling, we show that the framework efficiently learns an interpretable model, with higher modeling accuracy than existing methods.

The internal structure of the model depends on the type of the There is a variety of approaches that can be used to learning algorithm, so complex data-driven models can consist identify the optimal design of the data-driven model. For of several semi-independent blocks - this approach is usually instance, AutoML solutions can be based on random search referred to as ensembling [2]. There are several techniques to [5], Bayesian optimisation [6], reinforcement learning (RL) build complex models: for example, blending allows creating [7], Monte Carlo tree search [8], sequential model-based single-level ensembles of machine learning (ML) models, and optimization [9], gradient-based approaches [10]. However, stacking allows creating multi-level ones. Other approaches are most of them are less flexible than evolutionary approaches to based on the representation of a model structure (or even the the model design (implemented e.g. in [11]). Their conceptual whole modeling pipeline) as a directed acyclic graph (DAG).