A major factor in the success of deep neural networks is the use of sophisticated architectures rather than the classical multilayer perceptron (MLP). Residual networks (ResNets) stand out among these powerful modern architectures. Previous works focused on the optimization advantages of deep ResNets over deep MLPs. In this paper, we show another distinction between the two models, namely, a tendency of ResNets to promote smoother interpolations than MLPs. We analyze this phenomenon via the neural tangent kernel (NTK) approach. First, we compute the NTK for a considered ResNet model and prove its stability during gradient descent training. Then, we show by various evaluation methodologies that the NTK of ResNet, and its kernel regression results, are smoother than the ones of MLP. The better smoothness observed in our analysis may explain the better generalization ability of ResNets and the practice of moderately attenuating the residual blocks.

In this paper, we focus on developing a novel unsupervised machine learning algorithm, named graph based multi-layer k-means++ (G-MLKM), to solve data-target association problem when targets move on a constrained space and minimal information of the targets can be obtained by sensors. Instead of employing the traditional data-target association methods that are based on statistical probabilities, the G-MLKM solves the problem via data clustering. We first will develop the Multi-layer K-means++ (MLKM) method for data-target association at local space given a simplified constrained space situation. Then a p-dual graph is proposed to represent the general constrained space when local spaces are interconnected. Based on the dual graph and graph theory, we then generalize MLKM to G-MLKM by first understanding local data-target association and then extracting cross-local data-target association mathematically analyze the data association at intersections of that space. To exclude potential data-target association errors that disobey physical rules, we also develop error correction mechanisms to further improve the accuracy. Numerous simulation examples are conducted to demonstrate the performance of G-MLKM.

In this paper, we propose a one-stage online clustering method called Contrastive Clustering (CC) which explicitly performs the instance- and cluster-level contrastive learning. To be specific, for a given dataset, the positive and negative instance pairs are constructed through data augmentations and then projected into a feature space. Therein, the instance- and cluster-level contrastive learning are respectively conducted in the row and column space by maximizing the similarities of positive pairs while minimizing those of negative ones. Our key observation is that the rows of the feature matrix could be regarded as soft labels of instances, and accordingly the columns could be further regarded as cluster representations. By simultaneously optimizing the instance- and cluster-level contrastive loss, the model jointly learns representations and cluster assignments in an end-to-end manner. Extensive experimental results show that CC remarkably outperforms 17 competitive clustering methods on six challenging image benchmarks. In particular, CC achieves an NMI of 0.705 (0.431) on the CIFAR-10 (CIFAR-100) dataset, which is an up to 19\% (39\%) performance improvement compared with the best baseline.

Hierarchical clustering is an important technique to organize big data for exploratory data analysis. However, existing one-size-fits-all hierarchical clustering methods often fail to meet the diverse needs of different users. To address this challenge, we present an interactive steering method to visually supervise constrained hierarchical clustering by utilizing both public knowledge (e.g., Wikipedia) and private knowledge from users. The novelty of our approach includes 1) automatically constructing constraints for hierarchical clustering using knowledge (knowledge-driven) and intrinsic data distribution (data-driven), and 2) enabling the interactive steering of clustering through a visual interface (user-driven). Our method first maps each data item to the most relevant items in a knowledge base. An initial constraint tree is then extracted using the ant colony optimization algorithm. The algorithm balances the tree width and depth and covers the data items with high confidence. Given the constraint tree, the data items are hierarchically clustered using evolutionary Bayesian rose tree. To clearly convey the hierarchical clustering results, an uncertainty-aware tree visualization has been developed to enable users to quickly locate the most uncertain sub-hierarchies and interactively improve them. The quantitative evaluation and case study demonstrate that the proposed approach facilitates the building of customized clustering trees in an efficient and effective manner.

Learning feature space node embeddings that encode the position of a node within the context of a graph is useful in several graph prediction tasks. Majority of the existing graph neural networks (GNN) learn node embeddings that encode their local neighborhoods but not their positions. Consequently, two nodes that are vastly distant but located in similar local neighborhoods would map to similar embeddings. This limitation may prevent accurate performance in predictive tasks that rely on position information. In this paper, we address this gap by developing GraphReach, a position-aware, inductive GNN. GraphReach captures the global positions of nodes though reachability estimations with respect to a set of nodes called anchors. The reachability estimations compute the frequency with which a node may visit an anchor through any possible path. The anchors are strategically selected so that the reachability estimations across all nodes are maximized. We show that this combinatorial anchor selection problem is NP-hard and consequently, develop a greedy (1-1/e) approximation. An extensive experimental evaluation covering six datasets and five state-of-the-art GNN architectures reveal that GraphReach is consistently superior and provides up to 40% relative improvement in the predictive tasks of link prediction and pairwise node classification. In addition, GraphReach is more robust against adversarial attacks.

Matrix and tensor decomposition is the multiplication of a number of smaller matrices or tensors that are approximately disassembled by matrix and tensor. Up to now, the main matrix decomposition techniques have been widely used in computer vision, recommendation system, signal processing and other fields. Currently, standard methods for thirdorder nonnegative tensor decomposition include CP decomposition[1] and Tucker decomposition[2]. It's well known the normal nonnegative Tucker and CP tensor decomposition include non-convex optimization and that the global convergence is not guaranteed. One direction is to apply additional assumptions on data, such as a bounded variance, to transform the non-convex optimization problem into a convex one[3, 4]. Legendre decomposition[5] is a new nonnegative tensor decomposition method proposed by Mahito Sugiyama et al. Compared with the existing nonnegative tensor decomposition methods, the greatest contribution of Legendre decomposition lies in the transformation of the non-convex optimization problem onto a convex submanifold space without additional assumptions, which ensures global convergence, and the use of gradient descent can find a unique reconstructed tensor satisfying and the minimum Kullback-Leibler (KL) divergence from the input matrix. In this paper, we analyze Legendre tensor decomposition in both theory and application. From the perspective of theory, we aim to analyze the properties of dual parameters and dually flat manifold introduced in Legendre tensor decomposition.

The property of almost every point being a Lebesgue point has proven to be crucial for the consistency of several classification algorithms based on nearest neighbors. We characterize Lebesgue points in terms of a 1-Nearest Neighbor regression algorithm for pointwise estimation, fleshing out the role played by tie-breaking rules in the corresponding convergence problem. We then give an application of our results, proving the convergence of the risk of a large class of 1-Nearest Neighbor classification algorithms in general metric spaces where almost every point is a Lebesgue point.

Graph-structured data has recently received increasing attention in machine learning, with applications ranging from the prediction of chemical properties, e.g., whether a molecule is toxic [1], to the analysis of social network structures [2] and source code [3]. This paper focuses on the prediction of (global) graph properties, i.e., graph classification and regression. In order to predict a certain property of interest, for example to discriminate between graphs in a classification task, a learner must be able to detect, either explicitly or implicitly, characteristic features of a graph that are indicative of the sought property. To this end, suitable approaches have been developed in the fields of kernel-based machine learning and (deep) neural networks.

The rising availability of large volume data has enabled a wide application of statistical Machine Learning (ML) algorithms in the domains of Cyber-Physical Systems (CPS), Internet of Things (IoT) and Smart Building Networks (SBN). This paper proposes a learning-based framework for sequentially applying the data-driven statistical methods to predict indoor temperature and yields an algorithm for controlling building heating system accordingly. This framework consists of a two-stage modelling effort: in the first stage, an univariate time series model (AR) was employed to predict ambient conditions; together with other control variables, they served as the input features for a second stage modelling where an multivariate ML model (XGBoost) was deployed. The models were trained with real world data from building sensor network measurements, and used to predict future temperature trajectories. Experimental results demonstrate the effectiveness of the modelling approach and control algorithm, and reveal the promising potential of the data-driven approach in smart building applications over traditional dynamics-based modelling methods. By making wise use of IoT sensory data and ML algorithms, this work contributes to efficient energy management and sustainability in smart buildings.

We introduce explanatory guided learning (XGL), a novel interactive learning strategy in which a machine guides a human supervisor toward selecting informative examples for a classifier. The guidance is provided by means of global explanations, which summarize the classifier's behavior on different regions of the instance space and expose its flaws. Compared to other explanatory interactive learning strategies, which are machine-initiated and rely on local explanations, XGL is designed to be robust against cases in which the explanations supplied by the machine oversell the classifier's quality. Moreover, XGL leverages global explanations to open up the black-box of human-initiated interaction, enabling supervisors to select informative examples that challenge the learned model. By drawing a link to interactive machine teaching, we show theoretically that global explanations are a viable approach for guiding supervisors. Our simulations show that explanatory guided learning avoids overselling the model's quality and performs comparably or better than machine- and human-initiated interactive learning strategies in terms of model quality.