Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent baseline for variance reduction which fully exploits the structural form of the stochastic policy itself and does not make any additional assumptions about the MDP. We demonstrate and quantify the benefit of the action-dependent baseline through both theoretical analysis as well as numerical results, including an analysis of the suboptimality of the optimal state-dependent baseline. The result is a computationally efficient policy gradient algorithm, which scales to high-dimensional control problems, as demonstrated by a synthetic 2000-dimensional target matching task. Our experimental results indicate that action-dependent baselines allow for faster learning on standard reinforcement learning benchmarks and high-dimensional hand manipulation and synthetic tasks. Finally, we show that the general idea of including additional information in baselines for improved variance reduction can be extended to partially observed and multi-agent tasks.

Artificial intelligence might arguably be the newest frontier of human experience, but there's no denying that man has been fascinated with the concept for millennia. From the mythical stories of Hephaestus creating mechanical servants and brazen-footed bulls that puffed fire from their mouths, to the talking heads of the 13th century, to IBM Watson and modern forms of AI, the subject has been bubbling on the surface of human consciousness. The time is now here for AI to come of age; and, in many ways, it already has. But now there's a new problem, and it's not one of how AI can be implemented, as has been the major challenge in the past. AI has now sprouted into a plethora of forms, each rivaling the other in an attempt to showcase its superior capabilities.

CableLabs just released a cool short film called The Near Future: A Better Place that explores how emerging technologies in healthcare will transform our daily lives. A substantial percentage of the population worldwide is over the age of 60, and it will dramatically increase in the next two decades. This really underscores the importance of healthcare advancements, and connectivity is the underlying component that will power the emerging technologies that can transform our daily lives, such as IoT, telemedicine, intelligent agents and new sensors. For example, Cookie – the little robot AI Agent in the film is an in-home companion that provides social interaction, around the clock monitoring, as well as a direct interface with the complex system of care at the hospital. With this short film, CableLabs wants to inspire you and the entire tech and healthcare industry to help make this vision a reality in the near future.

DEPSO [1], or called DEPS, is an algorithm for (constrained) numerical optimization problem (NOP). DEPSO combines the advantages of Particle Swarm Optimization (PSO) and Differential Evolution (DE). It is incorporated into cooperative group optimization (CGO) system [2]. The DEPSO paper has been cited over 400 times with various applications. DEPSO was also implemented (by Sun Microsystems Inc.) into NLPSolver (Solver for Nonlinear Programming), an extension of Calc in Apache OpenOffice.

This paper continues the research that considers a new cognitive model based strongly on the human brain. In particular, it considers the neural binding structure of an earlier paper. It also describes some new methods in the areas of image processing and behaviour simulation. The work is all based on earlier research by the author and the new additions are intended to fit in with the overall design. For image processing, a grid-like structure is used with 'full linking'. Each cell in the classifier grid stores a list of all other cells it gets associated with and this is used as the learned image that new input is compared to. For the behaviour metric, a new prediction equation is suggested, as part of a simulation, that uses feedback and history to dynamically determine its course of action. While the new methods are from widely different topics, both can be compared with the binary-analog type of interface that is the main focus of the paper. It is suggested that the simplest of linking between a tree and ensemble can explain neural binding and variable signal strengths.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We'll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!): Let us know if you have suggestions for next week, and enjoy today's videos. We were at the 2018 Human Robot Interaction conference all this week, and on Wednesday, there was a special video session. The audience, who was provided with popcorn, voted by applause, and here are the top three videos.

Environmental monitoring allows authorities to understand the impact of potentially harmful phenomena, such as air pollution, excessive noise, and radiation. Recently, there has been considerable interest in participatory sensing as a paradigm for such large-scale data collection because it is cost-effective and able to capture more fine-grained data than traditional approaches that use stationary sensors scattered in cities. In this approach, ordinary citizens (non-expert contributors) collect environmental data using low-cost mobile devices. However, these participants are generally self-interested actors that have their own goals and make local decisions about when and where to take measurements. This can lead to highly inefficient outcomes, where observations are either taken redundantly or do not provide sufficient information about key areas of interest. To address these challenges, it is necessary to guide and to coordinate participants, so they take measurements when it is most informative. To this end, we develop a computationally-efficient coordination algorithm (adaptive Best-Match) that suggests to users when and where to take measurements. Our algorithm exploits probabilistic knowledge of human mobility patterns, but explicitly considers the uncertainty of these patterns and the potential unwillingness of people to take measurements when requested to do so. In particular, our algorithm uses a local search technique, clustering and random simulations to map participants to measurements that need to be taken in space and time. We empirically evaluate our algorithm on a real-world human mobility and air quality dataset and show that it outperforms the current state of the art by up to 24% in terms of utility gained.

Increasing energy efficiency in buildings can reduce costs and emissions substantially. Historically, this has been treated as a local, or single-agent, optimization problem. However, many buildings utilize the same types of thermal equipment e.g. electric heaters and hot water vessels. During operation, occupants in these buildings interact with the equipment differently thereby driving them to diverse regions in the state-space. Reinforcement learning agents can learn from these interactions, recorded as sensor data, to optimize the overall energy efficiency. However, if these agents operate individually at a household level, they can not exploit the replicated structure in the problem. In this paper, we demonstrate that this problem can indeed benefit from multi-agent collaboration by making use of targeted exploration of the state-space allowing for better generalization. We also investigate trade-offs between integrating human knowledge and additional sensors. Results show that savings of over 40% are possible with collaborative multi-agent systems making use of either expert knowledge or additional sensors with no loss of occupant comfort. We find that such multi-agent systems comfortably outperform comparable single agent systems.

We propose practical extensions to Bayesian optimization for solving dynamic problems. We model dynamic objective functions using spatiotemporal Gaussian process priors which capture all the instances of the functions over time. Our extensions to Bayesian optimization use the information learnt from this model to guide the tracking of a temporally evolving minimum. By exploiting temporal correlations, the proposed method also determines when to make evaluations, how fast to make those evaluations, and it induces an appropriate budget of steps based on the available information. Lastly, we evaluate our technique on synthetic and real-world problems.

Optimal Transport distances (also known as earth mover's distances or Wasserstein distances) design an optimal plan to move "mass" from one probability distribution to another. This problem can be traced back to the early work of Monge [1] and Kantorovich [2] and has been of constant interest for allowing natural formulations to the problems of comparing, interpolating, and measuring distances of functions [3]. On the other hand, computational optimal transport has gain popularity for its applications in learning theory [4], computer vision [5], computer graphics [6], statistical inference [7], information fusion [8]; and its relative complexity advantages with respect to classical methods [9]. Particularly, large-scale optimal transport has been of recent interest for the latest applications where large quantities of data are available and efficient algorithms are required [10, 11, 12]. Comprehensive accounts of the optimal transport problem and its computational aspects can be found in [13, 14, 15, 3]. One of the common uses of the Wasserstein distance is the aggregation of distributions by considering their barycenter [16], which itself is another distribution [17]. Wasserstein Barycenters has been shown superior to traditional Euclidean-based methods in a range of application such as image processing [16], economics and finance [18] and condensed matter physics [19]. Figure 1 shows a sample of 100 images of the digit 7 from the MNIST dataset [20] and their respective Euclidean mean and Wasserstein mean.