Efficient exploration remains a major challenge for reinforcement learning. One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic. Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropriately account for heteroscedasticity, even in the bandit setting. Motivated by recent findings that address this issue in bandits, we propose to use Information-Directed Sampling (IDS) for exploration in reinforcement learning. As our main contribution, we build on recent advances in distributional reinforcement learning and propose a novel, tractable approximation of IDS for deep Q-learning. The resulting exploration strategy explicitly accounts for both parametric uncertainty and heteroscedastic observation noise. We evaluate our method on Atari games and demonstrate a significant improvement over alternative approaches.

Abstract--Hybrid LSTM-fully convolutional networks (LSTM-FCN) for time series classification have produced state-of-the-art classification results on univariate time series. We show that replacing the LSTM with a gated recurrent unit (GRU) to create a GRU-fully convolutional network hybrid model (GRU-FCN) can offer even better performance on many time series datasets. The proposed GRU-FCN model outperforms state-of-the-art classification performance in many univariate and multivariate time series datasets. In addition, since the GRU uses a simpler architecture than the LSTM, it has fewer training parameters, less training time, and a simpler hardware implementation, compared to the LSTM-based models. A time series (TS) is a sequence of data points obtained at successive equally-spaced time points, ordinarily in a uniform interval time domain [1].

Smart mobility management would be an important prerequisite for future fog computing systems. In this research, we propose a learning-based handover optimization for the Internet of Vehicles that would assist the smooth transition of device connections and offloaded tasks between fog nodes. To accomplish this, we make use of machine learning algorithms to learn from vehicle interactions with fog nodes. Our approach uses a three-layer feed-forward neural network to predict the correct fog node at a given location and time with 99.2 % accuracy on a test set. We also implement a dual stacked recurrent neural network (RNN) with long short-term memory (LSTM) cells capable of learning the latency, or cost, associated with these service requests. We create a simulation in JAMScript using a dataset of real-world vehicle movements to create a dataset to train these networks. We further propose the use of this predictive system in a smarter request routing mechanism to minimize the service interruption during handovers between fog nodes and to anticipate areas of low coverage through a series of experiments and test the models' performance on a test set.

The literature about history matching is vast and despite the impressive number of methods proposed and the significant progresses reported in the last decade, conditioning reservoir models to dynamic data is still a challenging task. Ensemble-based methods are among the most successful and efficient techniques currently available for history matching. These methods are usually able to achieve reasonable data matches, especially if an iterative formulation is employed. However, they sometimes fail to preserve the geological realism of the model, which is particularly evident in reservoir with complex facies distributions. This occurs mainly because of the Gaussian assumptions inherent in these methods. This fact has encouraged an intense research activity to develop parameterizations for facies history matching. Despite the large number of publications, the development of robust parameterizations for facies remains an open problem. Deep learning techniques have been delivering impressive results in a number of different areas and the first applications in data assimilation in geoscience have started to appear in literature. The present paper reports the current results of our investigations on the use of deep neural networks towards the construction of a continuous parameterization of facies which can be used for data assimilation with ensemble methods. Specifically, we use a convolutional variational autoencoder and the ensemble smoother with multiple data assimilation. We tested the parameterization in three synthetic history-matching problems with channelized facies. We focus on this type of facies because they are among the most challenging to preserve after the assimilation of data. The parameterization showed promising results outperforming previous methods and generating well-defined channelized facies.

This work develops a deep learning power control framework for energy efficiency maximization in wireless interference networks. Rather than relying on suboptimal power allocation policies, the training of the deep neural network is based on the globally optimal power allocation rule, leveraging a newly proposed branch-and-bound procedure with a complexity affordable for the offline generation of large training sets. In addition, no initial power vector is required as input of the proposed neural network architecture, which further reduces the overall complexity. As a benchmark, we also develop a first-order optimal power allocation algorithm. Numerical results show that the neural network solution is virtually optimal, outperforming the more complex first-order optimal method, while requiring an extremely small online complexity.

Abstract--This work investigates the use of deep learning to perform user-cell association for sum-rate maximization in Massive MIMO networks. It is shown how a deep neural network can be trained to approach the optimal association rule with a much more limited computational complexity, thus enabling to update the association rule in real-time, on the basis of the mobility patterns of users. In particular, the proposed neural network design requires as input only the users' geographical positions. Numerical results show that it guarantees the same performance of traditional optimization-oriented methods. I. INTRODUCTION 5G wireless networks are scheduled to be rolled-out in only a couple of years.

Digital transformations are slated to transform the industry by reducing expenditures, improving operations, and providing a granular view of workflows enabling more effective decision-making. In the heart of all these digitization efforts in our industry lies machine learning. Machine learning enables us to build complex models on the data collected, leading to better decisions. In the simplest terms, it is a form of artificial intelligence (AI) which is designed to learn on its own or become better as it is fed more data. These algorithms have the potential to revolutionize our workflow in the future when the applicability of AI increases.

Spiking neural networks (SNNs) possess energy-efficient potential due to event-based computation. However, supervised training of SNNs remains a challenge as spike activities are non-differentiable. Previous SNNs training methods can basically be categorized into two classes, backpropagation-like training methods and plasticity-based learning methods. The former methods are dependent on energy-inefficient real-valued computation and non-local transmission, as also required in artificial neural networks (ANNs), while the latter either be considered biologically implausible or exhibit poor performance. Hence, biologically plausible (bio-plausible) high-performance supervised learning (SL) methods for SNNs remain deficient. In this paper, we proposed a novel bio-plausible SNN model for SL based on the symmetric spike-timing dependent plasticity (sym-STDP) rule found in neuroscience. By combining the sym-STDP rule with bio-plausible synaptic scaling and intrinsic plasticity of the dynamic threshold, our SNN model implemented SL well and achieved good performance in the benchmark recognition task (MNIST). To reveal the underlying mechanism of our SL model, we visualized both layer-based activities and synaptic weights using the t-distributed stochastic neighbor embedding (t-SNE) method after training and found that they were well clustered, thereby demonstrating excellent classification ability. As the learning rules were bio-plausible and based purely on local spike events, our model could be easily applied to neuromorphic hardware for online training and may be helpful for understanding SL information processing at the synaptic level in biological neural systems.

The manufacturing sector is heavily influenced by artificial intelligence-based technologies with the extraordinary increases in computational power and data volumes. It has been reported that 35% of US manufacturers are currently collecting data from sensors for manufacturing processes enhancement. Nevertheless, many are still struggling to achieve the 'Industry 4.0', which aims to achieve nearly 50% reduction in maintenance cost and total machine downtime by proper health management. For increasing productivity and reducing operating costs, a central challenge lies in the detection of faults or wearing parts in machining operations. Here we propose a data-driven, end-to-end framework for monitoring of manufacturing systems. This framework, derived from deep learning techniques, evaluates fused sensory measurements to detect and even predict faults and wearing conditions. This work exploits the predictive power of deep learning to extract hidden degradation features from noisy data. We demonstrate the proposed framework on several representative experimental manufacturing datasets drawn from a wide variety of applications, ranging from mechanical to electrical systems. Results reveal that the framework performs well in all benchmark applications examined and can be applied in diverse contexts, indicating its potential for use as a critical corner stone in smart manufacturing.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We'll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!): Let us know if you have suggestions for next week, and enjoy today's videos. The bar has now been set for robot holiday videos, thanks to FZI. Still waiting for a robot with a cookie to show up at my door.