Exploration bonus derived from the novelty of the states in an environment has become a popular approach to motivate exploration for deep reinforcement learning agents in the past few years. Recent methods such as curiosity-driven exploration usually estimate the novelty of new observations by the prediction errors of their system dynamics models. Due to the capacity limitation of the models and difficulty of performing next-frame prediction, however, these methods typically fail to balance between exploration and exploitation in high-dimensional observation tasks, resulting in the agents forgetting the visited paths and exploring those states repeatedly. Such inefficient exploration behavior causes significant performance drops, especially in large environments with sparse reward signals. In this paper, we propose to introduce the concept of optical flow estimation from the field of computer vision to deal with the above issue. We propose to employ optical flow estimation errors to examine the novelty of new observations, such that agents are able to memorize and understand the visited states in a more comprehensive fashion. We compare our method against the previous approaches in a number of experimental experiments. Our results indicate that the proposed method appears to deliver superior and long-lasting performance than the previous methods. We further provide a set of comprehensive ablative analysis of the proposed method, and investigate the impact of optical flow estimation on the learning curves of the DRL agents.

The problem of high-dimensional and large-scale representation of visual data is addressed from an unsupervised learning perspective. The emphasis is put on discrete representations, where the description length can be measured in bits and hence the model capacity can be controlled. The algorithmic infrastructure is developed based on the synthesis and analysis prior models whose rate-distortion properties, as well as capacity vs. sample complexity trade-offs are carefully optimized. These models are then extended to multi-layers, namely the RRQ and the ML-STC frameworks, where the latter is further evolved as a powerful deep neural network architecture with fast and sample-efficient training and discrete representations. For the developed algorithms, three important applications are developed. First, the problem of large-scale similarity search in retrieval systems is addressed, where a double-stage solution is proposed leading to faster query times and shorter database storage. Second, the problem of learned image compression is targeted, where the proposed models can capture more redundancies from the training images than the conventional compression codecs. Finally, the proposed algorithms are used to solve ill-posed inverse problems. In particular, the problems of image denoising and compressive sensing are addressed with promising results.

Dynamic portfolio optimization is the process of sequentially allocating wealth to a collection of assets in some consecutive trading periods, based on investors' return-risk profile. Automating this process with machine learning remains a challenging problem. Here, we design a deep reinforcement learning (RL) architecture with an autonomous trading agent such that, investment decisions and actions are made periodically, based on a global objective, with autonomy. In particular, without relying on a purely model-free RL agent, we train our trading agent using a novel RL architecture consisting of an infused prediction module (IPM), a generative adversarial data augmentation module (DAM) and a behavior cloning module (BCM). Our model-based approach works with both on-policy or off-policy RL algorithms. We further design the back-testing and execution engine which interact with the RL agent in real time. Using historical {\em real} financial market data, we simulate trading with practical constraints, and demonstrate that our proposed model is robust, profitable and risk-sensitive, as compared to baseline trading strategies and model-free RL agents from prior work.

A graph is a powerful concept for representation of relations between pairs of entities. Data with underlying graph structure can be found across many disciplines and there is a natural desire for understanding such data better. Deep learning (DL) has achieved significant breakthroughs in a variety of machine learning tasks in recent years, especially where data is structured on a grid, such as in text, speech, or image understanding. However, surprisingly little has been done to explore the applicability of DL on arbitrary graph-structured data directly. The goal of this thesis is to investigate architectures for DL on graphs and study how to transfer, adapt or generalize concepts that work well on sequential and image data to this domain. We concentrate on two important primitives: embedding graphs or their nodes into a continuous vector space representation (encoding) and, conversely, generating graphs from such vectors back (decoding). To that end, we make the following contributions. First, we introduce Edge-Conditioned Convolutions (ECC), a convolution-like operation on graphs performed in the spatial domain where filters are dynamically generated based on edge attributes. The method is used to encode graphs with arbitrary and varying structure. Second, we propose SuperPoint Graph, an intermediate point cloud representation with rich edge attributes encoding the contextual relationship between object parts. Based on this representation, ECC is employed to segment large-scale point clouds without major sacrifice in fine details. Third, we present GraphVAE, a graph generator allowing us to decode graphs with variable but upper-bounded number of nodes making use of approximate graph matching for aligning the predictions of an autoencoder with its inputs. The method is applied to the task of molecule generation.

In this survey paper, we systematically summarize the current literature on studies that apply machine learning (ML) and data mining techniques to bearing fault diagnostics. Conventional ML methods, including artificial neural network (ANN), principal component analysis (PCA), support vector machines (SVM), etc., have been successfully applied to detecting and categorizing bearing faults since the last decade, while the application of deep learning (DL) methods has sparked great interest in both the industry and academia in the last five years. In this paper, we will first review the conventional ML methods, before taking a deep dive into the latest developments in DL algorithms for bearing fault applications. Specifically, the superiority of the DL based methods over the conventional ML methods are analyzed in terms of metrics directly related to fault feature extraction and classifier performances; the new functionalities offered by DL techniques that cannot be accomplished before are also summarized. In addition, to obtain a more intuitive insight, a comparative study is performed on the classifier performance and accuracy for a number of papers utilizing the open source Case Western Reserve University (CWRU) bearing data set. Finally, based on the nature of the time-series 1-D data obtained from sensors monitoring the bearing conditions, recommendations and suggestions are provided to applying DL algorithms on bearing fault diagnostics based on specific applications, as well as future research directions to further improve its performance.

Intel is back with another RealSense camera, but this one has a slight twist: it's meant to give machines a sense of place. The lengthily-titled RealSense Tracking Camera T265 uses inside-out tracking (that is, it doesn't need outside sensors) to help localize robots and other autonomous machines, particularly in situations where GPS is unreliable or non-existent. A farming robot, for instance, could both map a field as well as adapt on the fly to obstacles like buildings and rocks. It's relying on the same Myriad 2 processing hardware seen in other recent projects, which takes much of the processing burden away from other devices without heavy energy demands. The only requirements are 1.5W of power, a USB connection and enough memory to power it up. You could stick this on a drone, to put it another way.

Amazon is selling a robotic toilet cleaner for $499.99 (£400) that will keep the lavatory squeaky clean for you. The droid is called Giddel and is designed to scrub the rim and bowl of a toilet so customers don't have to complete the arduous chore. It weighs more than six pounds (3kg) and is powered by a rechargeable lithium-ion battery. The hefty price tag includes the robot, cleaning accessories and an elongated toilet seat to allow the robot to clip on. Amazon is selling a robotic toilet cleaner called Giddel (pictured) for $499.99 (£400) that will keep the lavatory squeaky clean The droid is designed to scrub the rim and bowl of a toilet so customers don't have to complete the arduous chore It weighs 1.5 pounds (3kg) and is powered by a rechargeable lithium-ion battery.

The entire website of Elon Musk's private charitable foundation is shorter than many of the Tesla CEO's contentious tweets. Grants are made in support of: Renewable energy research and advocacy; Human space exploration research and advocacy; Pediatric research; Science and engineering education," the site reads. Documents obtained by the Guardian reveal how the foundation has put that vague mission statement into practice. Together, the documents show that many of the organization's donations have gone far beyond its stated scope. Some have benefited the billionaire's own family and initiatives, others have tackled his pet peeves – the foundation has given more money to Musk's own artificial intelligence research than to any of the more traditional charities it says it supports.

What does a nuclear power plant disaster have to do with machine learning? The safety plan for Fukushima Daiichi nuclear power plant was designed using the historical data for past 400 years. The structural engineers designed the plant to withstand an earthquake of 8.6 intensity on Richter scale and a tsunami as high as 5.7 meters. These threshold numbers were decided using predictive modeling. So, they had the data for earthquakes(intensity and annual frequency) in last 400 years and they were looking for a model that can help predict the earthquakes in future.

In this paper, we propose a new method to tackle the mapping challenge from time-series data to spatial image in the field of seismic exploration, i.e., reconstructing the velocity model directly from seismic data by deep neural networks (DNNs). The conventional way to address this ill-posed seismic inversion problem is through iterative algorithms, which suffer from poor nonlinear mapping and strong non-uniqueness. Other attempts may either import human intervention errors or underuse seismic data. The challenge for DNNs mainly lies in the weak spatial correspondence, the uncertain reflection-reception relationship between seismic data and velocity model as well as the time-varying property of seismic data. To approach these challenges, we propose an end-to-end Seismic Inversion Networks (SeisInvNet for short) with novel components to make the best use of all seismic data. Specifically, we start with every seismic trace and enhance it with its neighborhood information, its observation setup and global context of its corresponding seismic profile. Then from enhanced seismic traces, the spatially aligned feature maps can be learned and further concatenated to reconstruct velocity model. In general, we let every seismic trace contribute to the reconstruction of the whole velocity model by finding spatial correspondence. The proposed SeisInvNet consistently produces improvements over the baselines and achieves promising performance on our proposed SeisInv dataset according to various evaluation metrics, and the inversion results are more consistent with the target from the aspects of velocity value, subsurface structure and geological interface. In addition to the superior performance, the mechanism is also carefully discussed, and some potential problems are identified for further study.