Non-stationary environments are challenging for reinforcement learning algorithms. If the state transition and/or reward functions change based on latent factors, the agent is effectively tasked with optimizing a behavior that maximizes performance over a possibly infinite random sequence of Markov Decision Processes (MDPs), each of which drawn from some unknown distribution. We call each such MDP a context. Most related works make strong assumptions such as knowledge about the distribution over contexts, the existence of pre-training phases, or a priori knowledge about the number, sequence, or boundaries between contexts. We introduce an algorithm that efficiently learns policies in non-stationary environments. It analyzes a possibly infinite stream of data and computes, in real-time, high-confidence change-point detection statistics that reflect whether novel, specialized policies need to be created and deployed to tackle novel contexts, or whether previously-optimized ones might be reused. We show that (i) this algorithm minimizes the delay until unforeseen changes to a context are detected, thereby allowing for rapid responses; and (ii) it bounds the rate of false alarm, which is important in order to minimize regret. Our method constructs a mixture model composed of a (possibly infinite) ensemble of probabilistic dynamics predictors that model the different modes of the distribution over underlying latent MDPs. We evaluate our algorithm on high-dimensional continuous reinforcement learning problems and show that it outperforms state-of-the-art (model-free and model-based) RL algorithms, as well as state-of-the-art meta-learning methods specially designed to deal with non-stationarity.

In this paper, by leveraging abundant observational transaction data, we propose a novel data-driven and interpretable pricing approach for markdowns, consisting of counterfactual prediction and multi-period price optimization. Firstly, we build a semi-parametric structural model to learn individual price elasticity and predict counterfactual demand. This semi-parametric model takes advantage of both the predictability of nonparametric machine learning model and the interpretability of economic model. Secondly, we propose a multi-period dynamic pricing algorithm to maximize the overall profit of a perishable product over its finite selling horizon. Different with the traditional approaches that use the deterministic demand, we model the uncertainty of counterfactual demand since it inevitably has randomness in the prediction process. Based on the stochastic model, we derive a sequential pricing strategy by Markov decision process, and design a two-stage algorithm to solve it. The proposed algorithm is very efficient. It reduces the time complexity from exponential to polynomial. Experimental results show the advantages of our pricing algorithm, and the proposed framework has been successfully deployed to the well-known e-commerce fresh retail scenario - Freshippo.

Learn to write programs using the foundational AI algorithms powering everything from NASA's Mars Rover to DeepMind's AlphaGo Zero. Learn to write AI programs using the algorithms powering everything from NASA's Mars Rover to DeepMind's AlphaGo Zero.

User-generated data on social media contain rich information about who we are, what we like and how we make decisions. In this paper, we survey representative work on learning a concise latent user representation (a.k.a. user embedding) that can capture the main characteristics of a social media user. The learned user embeddings can later be used to support different downstream user analysis tasks such as personality modeling, suicidal risk assessment and purchase decision prediction. The temporal nature of user-generated data on social media has largely been overlooked in much of the existing user embedding literature. In this survey, we focus on research that bridges the gap by incorporating temporal/sequential information in user representation learning. We categorize relevant papers along several key dimensions, identify limitations in the current work and suggest future research directions.

The framework is founded on an approach of minimising the Kullback-Leibler (KL) divergence between a variational solution and the posterior, in a constrained space. The works (Dudík et al., 2004, 2007; Altun and Smola, 2006) first raised the idea of including constraints in maximum entropy density estimation and provided a theoretical analysis. Based on convex duality theory, the optimal solution of the regularised posterior is found to be the original posterior of the model, discounted by the constrained pseudo likelihood introduced by the constraints. Later work founded on the idea of posterior constraints includes (Graça et al., 2009) which proposed constraining the E-step of an Expectation-maximization (EM) algorithm, in order to impose feature constraints on the solution.

I believe that the book describes itself quite well when it says: Mathematically correct yet highly intuitive…This book would be great for a class that one takes before one takes my statistical learning class. I often run into beginning graduate Data Science students whose background is not math (e.g., CS or Business) and they are not ready…The book fills an important niche, in that it provides a self-contained introduction to material that is useful for a higher-level statistical learning course. I think that it compares well with competing books, particularly in that it takes a more "Data Science" and "example driven" approach than more classical books." "This text by Matloff (Univ. of California, Davis) affords an excellent introduction to statistics for the data science student…Its examples are often drawn from data science applications such as hidden Markov models and remote sensing, to name a few… All the models and concepts are explained well in precise mathematical terms (not presented as formal proofs), to help students gain an intuitive understanding."

There is a growing interest in using electric vehicles (EVs) and drones for many applications. However, battery-oriented issues, including range anxiety and battery degradation, impede adoption. Battery swap stations are one alternative to reduce these concerns that allow the swap of depleted for full batteries in minutes. We consider the problem of deriving actions at a battery swap station when explicitly considering the uncertain arrival of swap demand, battery degradation, and replacement. We model the operations at a battery swap station using a finite horizon Markov Decision Process model for the stochastic scheduling, allocation, and inventory replenishment problem (SAIRP), which determines when and how many batteries are charged, discharged, and replaced over time. We present theoretical proofs for the monotonicity of the value function and monotone structure of an optimal policy for special SAIRP cases. Due to the curses of dimensionality, we develop a new monotone approximate dynamic programming (ADP) method, which intelligently initializes a value function approximation using regression. In computational tests, we demonstrate the superior performance of the new regression-based monotone ADP method as compared to exact methods and other monotone ADP methods. Further, with the tests, we deduce policy insights for drone swap stations.

There is considerable interest in designing meta-reinforcement learning (meta-RL) algorithms, which enable autonomous agents to adapt new tasks from small amount of experience. In meta-RL, the specification (such as reward function) of current task is hidden from the agent. In addition, states are hidden within each task owing to sensor noise or limitations in realistic environments. Therefore, the meta-RL agent faces the challenge of specifying both the hidden task and states based on small amount of experience. To address this, we propose estimating disentangled belief about task and states, leveraging an inductive bias that the task and states can be regarded as global and local features of each task. Specifically, we train a hierarchical state-space model (HSSM) parameterized by deep neural networks as an environment model, whose global and local latent variables correspond to task and states, respectively. Because the HSSM does not allow analytical computation of posterior distribution, i.e., belief, we employ amortized inference to approximate it. After the belief is obtained, we can augment observations of a model-free policy with the belief to efficiently train the policy. Moreover, because task and state information are factorized and interpretable, the downstream policy training is facilitated compared with the prior methods that did not consider the hierarchical nature. Empirical validations on a GridWorld environment confirm that the HSSM can separate the hidden task and states information. Then, we compare the meta-RL agent with the HSSM to prior meta-RL methods in MuJoCo environments, and confirm that our agent requires less training data and reaches higher final performance.

Curiosity is what drives organisms to investigate each other and their environment. It is considered by many to be as intrinsic as hunger and thirst, but the neurobiological mechanisms behind curiosity have remained elusive. In mice, Ahmadlou et al. found that a specific population of genetically identified γ-aminobutyric acid (GABA)—ergic neurons in a brain region called the zona incerta receive excitatory input in the form of novelty and/or arousal information from the prelimbic cortex, and these neurons send inhibitory projections to the periaqueductal gray region (see the Perspective by Farahbakhsh and Siciliano). This circuitry is necessary for the exploration of new objects and conspecifics. Science , this issue p. [eabe9681][1]; see also p. [684][2] ### INTRODUCTION Motivational drives are internal states that can be different even in similar interactions with external stimuli. Curiosity as the motivational drive for novelty-seeking and investigating the surrounding environment is for survival as essential and intrinsic as hunger. Curiosity, hunger, and appetitive aggression drive three different goal-directed behaviors—novelty seeking, food eating, and hunting—but these behaviors are composed of similar actions in animals. This similarity of actions has made it challenging to study novelty seeking and distinguish it from eating and hunting in nonarticulating animals. The brain mechanisms underlying this basic survival drive, curiosity, and novelty-seeking behavior have remained unclear. ### RATIONALE In spite of having well-developed techniques to study mouse brain circuits, there are many controversial and different results in the field of motivational behavior. This has left the functions of motivational brain regions such as the zona incerta (ZI) still uncertain. Not having a transparent, nonreinforced, and easily replicable paradigm is one of the main causes of this uncertainty. Therefore, we chose a simple solution to conduct our research: giving the mouse freedom to choose what it wants—double free-access choice. By examining mice in an experimental battery of object free-access double-choice (FADC) and social interaction tests—using optogenetics, chemogenetics, calcium fiber photometry, multichannel recording electrophysiology, and multicolor mRNA in situ hybridization—we uncovered a cell type–specific cortico-subcortical brain circuit of the curiosity and novelty-seeking behavior. ### RESULTS We analyzed the transitions within action sequences in object FADC and social interaction tests. Frequency and hidden Markov model analyses showed that mice choose different action sequences in interaction with novel objects and in early periods of interaction with novel conspecifics compared with interaction with familiar objects or later periods of interaction with conspecifics, which we categorized as deep and shallow investigation, respectively. This finding helped us to define a measure of depth of investigation that indicates how much a mouse prefers deep over shallow investigation and reflects the mouse’s motivational level to investigate, regardless of total duration of investigation. Optogenetic activation of inhibitory neurons in medial ZI (ZIm), ZImGAD2 neurons, showed a dramatic increase in positive arousal level, depth of investigation, and duration of interaction with conspecifics and novel objects compared with familiar objects, crickets, and food. Optogenetic or chemogenetic deactivation of these neurons decreased depth and duration of investigation. Moreover, we found that ZImGAD2 neurons are more active during deep investigation as compared with during shallow investigation. We found that activation of prelimbic cortex (PL) axons into ZIm increases arousal level, and chemogenetic deactivation of these axons decreases the duration and depth of investigation. Calcium fiber photometry of these axons showed no difference in activity between shallow and deep investigation, suggesting a nonspecific motivation. Optogenetic activation of ZImGAD2 axons into lateral periaqueductal gray (lPAG) increases the arousal level, whereas chemogenetic deactivation of these axons decreases duration and depth of investigation. Calcium fiber photometry of these axons showed high activity during deep investigation and no significant activity during shallow investigation, suggesting a thresholding mechanism. Last, we found a new subpopulation of inhibitory neurons in ZIm expressing tachykinin 1 (TAC1) that monosynaptically receive PL inputs and project to lPAG. Optogenetic activation and deactivation of these neurons, respectively, increased and decreased depth and duration of investigation. ### CONCLUSION Our experiments revealed different action sequences based on the motivational level of novelty seeking. Moreover, we uncovered a new brain circuit underlying curiosity and novelty-seeking behavior, connecting excitatory neurons of PL to lPAG through TAC1+ inhibitory neurons of ZIm. ![Figure][3] Brain mechanism of curiosity. ( A ) How we mapped motivational level to action sequences. ( B ) Experimental battery to distinguish novelty-seeking behavior from food eating and hunting in mice with photoactivation of ZImGAD2 neurons. ( C ) Schematic of calcium activity in PL→ZIm, ZIm, and ZIm→PAG during shallow and deep investigation. ( D ) TAC1+ neurons as a subpopulation of ZImGAD2 neurons receive input from PL and project to PAG. HMM, hidden Markov model. Exploring the physical and social environment is essential for understanding the surrounding world. We do not know how novelty-seeking motivation initiates the complex sequence of actions that make up investigatory behavior. We found in mice that inhibitory neurons in the medial zona incerta (ZIm), a subthalamic brain region, are essential for the decision to investigate an object or a conspecific. These neurons receive excitatory input from the prelimbic cortex to signal the initiation of exploration. This signal is modulated in the ZIm by the level of investigatory motivation. Increased activity in the ZIm instigates deep investigative action by inhibiting the periaqueductal gray region. A subpopulation of inhibitory ZIm neurons expressing tachykinin 1 (TAC1) modulates the investigatory behavior. [1]: /lookup/doi/10.1126/science.abe9681 [2]: /lookup/doi/10.1126/science.abi7270 [3]: pending:yes

In this paper, we develop a safe decision-making method for self-driving cars in a multi-lane, single-agent setting. The proposed approach utilizes deep reinforcement learning (RL) to achieve a high-level policy for safe tactical decision-making. We address two major challenges that arise solely in autonomous navigation. First, the proposed algorithm ensures that collisions never happen, and therefore accelerate the learning process. Second, the proposed algorithm takes into account the unobservable states in the environment. These states appear mainly due to the unpredictable behavior of other agents, such as cars, and pedestrians, and make the Markov Decision Process (MDP) problematic when dealing with autonomous navigation. Simulations from a well-known self-driving car simulator demonstrate the applicability of the proposed method