The ability to detect that something has changed in an environment is valuable, but often only if it can be accurately conveyed to a human operator. We introduce Viewpoint Invariant Change Captioning, and develop models which can both localize and describe via natural language complex changes in an environment. Moreover, we distinguish between a change in a viewpoint and an actual scene change (e.g. a change of objects' attributes). To study this new problem, we collect a Viewpoint Invariant Change Captioning Dataset (VICC), building it off the CLEVR dataset and engine. We introduce 5 types of scene changes, including changes in attributes, positions, etc. To tackle this problem, we propose an approach that distinguishes a viewpoint change from an important scene change, localizes the change between "before" and "after" images, and dynamically attends to the relevant visual features when describing the change. We benchmark a number of baselines on our new dataset, and systematically study the different change types. We show the superiority of our proposed approach in terms of change captioning and localization. Finally, we also show that our approach is general and can be applied to real images and language on the recent Spot-the-diff dataset.

Abstract: Deep learning has revolutionized many machine learning tasks in recent years, ranging from image classification and video processing to speech recognition and natural language understanding. The data in these tasks are typically represented in the Euclidean space. However, there is an increasing number of applications where data are generated from non-Euclidean domains and are represented as graphs with complex relationships and interdependency between objects. The complexity of graph data has imposed significant challenges on existing machine learning algorithms. Recently, many studies on extending deep learning approaches for graph data have emerged.

Machine learning on graphs is a difficult task due to the highly complex, but also informative graph structure. This post is the first in a series on how to do deep learning on graphs with Graph Convolutional Networks (GCNs), a powerful type of neural network designed to work directly on graphs and leverage their structural information. In this post, I will give an introduction to GCNs and illustrate how information is propagated through the hidden layers of a GCN using coding examples. We'll see how the GCN aggregates information from the previous layers and how this mechanism produces useful feature representations of nodes in graphs. GCNs are a very powerful neural network architecture for machine learning on graphs.

I find the activity of codifying and comparing various interpretations of a particular process in the pursuit of strengthening one's own interpretation of said process to be a worthy one. I have previously done so with alternate interpretations of what we could call the machine learning process (and which could reasonably be closely aligned with the data science or data mining processes, at least to some degree), of which you can find examples here and here and here. These previous posts have considered the classic CRISP-DM model, the KDD Process, Francois Chollet's 4 step model (aimed at Keras, but generalizable), Yufeng Guo's 7 steps to machine learning, and even modifications aimed specifically at more narrow disciplines, such as the text-based data science task framework. In an effort to further refine our internal models, this post will present an overview of Aurélien Géron's Machine Learning Project Checklist, as seen in his bestselling book, "Hands-On Machine Learning with Scikit-Learn & TensorFlow." It's a similar approach to that of, say, Guo's 7 step process, but at a subtly higher level; it's presented as a checklist of approaching projects, and so it feels less prescriptive and more descriptive, a reminder of what you should be doing as you do it as opposed to some grand explanation of why you are doing what you are doing.

The massive adoption of Internet of Things (IoT) opens a plethora of new use cases, applications, frameworks, and data processing architectures. A new ecosystem of supporting technologies is being developed in parallel with IoT to enable resource provisioning for resource‐constrained devices and systems (Baktir, Ozgovde, & Ersoy, 2017; Mao, You, Zhang, Huang, & Letaief, 2017; F. Wang, Hu, Hu, Zhou, & Zhao, 2017). The core of future IoT systems will be designed by integrating mobile edge computing systems, software‐defined networks, 5G, augmented reality, and data mining (including machine learning and artificial intelligence) to name a few (Baktir et al., 2017; Mao et al., 2017). Data mining is the process of discovering hidden knowledge patterns from raw data; therefore, the execution of knowledge discovery processes in IoT environments will leverage the utility of IoT systems. In essence, data mining will play a vital role in highly interactive and intelligent IoT systems.

The massive adoption of Internet of Things (IoT) opens a plethora of new use cases, applications, frameworks, and data processing architectures. A new ecosystem of supporting technologies is being developed in parallel with IoT to enable resource provisioning for resource‐constrained devices and systems (Baktir, Ozgovde, & Ersoy, 2017; Mao, You, Zhang, Huang, & Letaief, 2017; F. Wang, Hu, Hu, Zhou, & Zhao, 2017). The core of future IoT systems will be designed by integrating mobile edge computing systems, software‐defined networks, 5G, augmented reality, and data mining (including machine learning and artificial intelligence) to name a few (Baktir et al., 2017; Mao et al., 2017). Data mining is the process of discovering hidden knowledge patterns from raw data; therefore, the execution of knowledge discovery processes in IoT environments will leverage the utility of IoT systems. In essence, data mining will play a vital role in highly interactive and intelligent IoT systems.

Explanation in machine learning and related fields such as artificial intelligence aims at making machine learning models and their decisions understandable to humans. Existing work suggests that personalizing explanations might help to improve understandability. In this work, we derive a conceptualization of personalized explanation by defining and structuring the problem based on prior work on machine learning explanation, personalization (in machine learning) and concepts and techniques from other domains such as privacy and knowledge elicitation. We perform a categorization of explainee information used in the process of personalization as well as describing means to collect this information. We also identify three key explanation properties that are amendable to personalization: complexity, decision information and presentation. We also enhance existing work on explanation by introducing additional desiderata and measures to quantify the quality of personalized explanations.

Companies must refuse to create artificial intelligence that is unethical and could harm humanity, even if it affects their profits, a senior director at Microsoft UK has said. Hugh Milward, Senior Director of Corporate, External and Legal Affairs, said businesses need to "draw a line" on what is acceptable when developing cutting-edge technology and understand their responsibilities. "Just because something can be done, doesn't mean it should be done," he told the Tech UK Digital Ethics Summit in London. The event heard from leading figures in business and government on how the UK can remain a global leader in building digital services and technology, and discussed how they should be used for the benefits of everyone. "It is essential to build trust in technology such as AI," Milward said.

Deep learning has revolutionized many machine learning tasks in recent years, ranging from image classification and video processing to speech recognition and natural language understanding. The data in these tasks are typically represented in the Euclidean space. However, there is an increasing number of applications where data are generated from non-Euclidean domains and are represented as graphs with complex relationships and interdependency between objects. The complexity of graph data has imposed significant challenges on existing machine learning algorithms. Recently, many studies on extending deep learning approaches for graph data have emerged. In this survey, we provide a comprehensive overview of graph neural networks (GNNs) in data mining and machine learning fields. We propose a new taxonomy to divide the state-of-the-art graph neural networks into different categories. With a focus on graph convolutional networks, we review alternative architectures that have recently been developed; these learning paradigms include graph attention networks, graph autoencoders, graph generative networks, and graph spatial-temporal networks. We further discuss the applications of graph neural networks across various domains and summarize the open source codes and benchmarks of the existing algorithms on different learning tasks. Finally, we propose potential research directions in this fast-growing field.

Multi-output learning aims to simultaneously predict multiple outputs given an input. It is an important learning problem due to the pressing need for sophisticated decision making in real-world applications. Inspired by big data, the 4Vs characteristics of multi-output imposes a set of challenges to multi-output learning, in terms of the volume, velocity, variety and veracity of the outputs. Increasing number of works in the literature have been devoted to the study of multi-output learning and the development of novel approaches for addressing the challenges encountered. However, it lacks a comprehensive overview on different types of challenges of multi-output learning brought by the characteristics of the multiple outputs and the techniques proposed to overcome the challenges. This paper thus attempts to fill in this gap to provide a comprehensive review on this area. We first introduce different stages of the life cycle of the output labels. Then we present the paradigm on multi-output learning, including its myriads of output structures, definitions of its different sub-problems, model evaluation metrics and popular data repositories used in the study. Subsequently, we review a number of state-of-the-art multi-output learning methods, which are categorized based on the challenges.