In classification applications such as severe disease diagnosis and fraud detection, people have clear priorities over the two types of classification errors. For instance, diagnosing a patient with cancer to be healthy may lead to loss of life, which incurs a much higher cost than the other way around. The classical binary classification paradigm does not take into account such priorities, as it aims to minimize the overall classification error. In contrast, the Neyman-Pearson (NP) paradigm seeks classifiers with a minimal type II error while having the prioritized type I error constrained under a user-specified level, addressing asymmetric type I/II error priorities in the previously mentioned scenarios. Despite recent advances in the NP classification literature, two essential issues pose challenges: i) current theoretical framework assumes bounded feature support, which does not admit parametric settings; ii) in practice, existing NP classifiers involve splitting class 0 samples into two parts using a pre-fixed split proportion. To address the first challenge, we present NP-sLDA that adapts the popular sparse linear discriminant analysis (sLDA, Mai et al. (2012)) to the NP paradigm. On the theoretical front, this is the first theoretically justified NP classifier that takes parametric assumptions and unbounded feature support. We formulate a new conditional margin assumption and a new conditional detection condition to accommodate unbounded feature support and show that NP-sLDA satisfies the NP oracle inequalities. Numerical results show that NP-sLDA is a valuable addition to the existing NP classifiers. To address the second challenge, we construct a general data-adaptive sample splitting scheme that improves the classification performance upon the default half-half class 0 split used in Tong et al. (2018).

Find 16 colors out of millions of colors (24 bit to 4 bit reduction) which represents a picture best, using unsupervised learning algorithm K-means for clustering. You can see the result of my programming exercise at the top. Nice example to visualize how you can reduce highly dimensional problem spaces into something which a computer can handle better without losing the core information.

This problem also appeared as an assignment problem in the coursera online course Mathematics for Machine Learning: Multivariate Calculus. The description of the problem is taken from the assignment itself. In this assignment, we shall train a neural network to draw a curve. The curve takes one input variable, the amount traveled along the curve from 0 to 1, and returns 2 outputs, the 2D coordinates of the position of points on the curve. The below table shows the first few rows of the dataset.

The k-means is a simple algorithm that divides the data set into k partitions for n objects where k n. In this method, the data set is partitioned into homogeneous groups with similar characteristics. The similarity or dissimilarity is defined by calculating the distance between the centroid and data points. The clusters are formed when the optimization function for the algorithm achieves its objective -- less intracluster distances and more intercluster distances. Repeat steps 3 and 4 until the centroids in clusters change.

Logistic Regression is one of the most used Machine Learning algorithms for binary classification. It is a simple Algorithm that you can use as a performance baseline, it is easy to implement and it will do well enough in many tasks. Therefore every Machine Learning engineer should be familiar with its concepts. The building block concepts of Logistic Regression can also be helpful in deep learning while building neural networks. In this post, you will learn what Logistic Regression is, how it works, what are advantages and disadvantages and much more.

In this work, we propose a new training method for finding minimum weight norm solutions in over-parameterized neural networks (NNs). This method seeks to improve training speed and generalization performance by framing NN training as a constrained optimization problem wherein the sum of the norm of the weights in each layer of the network is minimized, under the constraint of exactly fitting training data. It draws inspiration from support vector machines (SVMs), which are able to generalize well, despite often having an infinite number of free parameters in their primal form, and from recent theoretical generalization bounds on NNs which suggest that lower norm solutions generalize better. To solve this constrained optimization problem, our method employs Lagrange multipliers that act as integrators of error over training and identify `support vector'-like examples. The method can be implemented as a wrapper around gradient based methods and uses standard back-propagation of gradients from the NN for both regression and classification versions of the algorithm. We provide theoretical justifications for the effectiveness of this algorithm in comparison to early stopping and $L_2$-regularization using simple, analytically tractable settings. In particular, we show faster convergence to the max-margin hyperplane in a shallow network (compared to vanilla gradient descent); faster convergence to the minimum-norm solution in a linear chain (compared to $L_2$-regularization); and initialization-independent generalization performance in a deep linear network. Finally, using the MNIST dataset, we demonstrate that this algorithm can boost test accuracy and identify difficult examples in real-world datasets.

This note considers softmax parameter estimation when little/no labeled training data is available, but a priori information about the relative geometry of class label log-odds boundaries is available. It is shown that `data-free' softmax model synthesis corresponds to solving a linear system of parameter equations, wherein desired dominant class log-odds boundaries are encoded via convex polytopes that decompose the input feature space. When solvable, the linear equations yield closed-form softmax parameter solution families using class boundary polytope specifications only. This allows softmax parameter learning to be implemented without expensive brute force data sampling and numerical optimization. The linear equations can also be adapted to constrained maximum likelihood estimation in data-sparse settings. Since solutions may also fail to exist for the linear parameter equations derived from certain polytope specifications, it is thus also shown that there exist probabilistic classification problems over m convexly separable classes for which the log-odds boundaries cannot be learned using an m-class softmax model.

In order to scale standard Gaussian process (GP) regression to large-scale datasets, aggregation models employ factorized training process and then combine predictions from distributed experts. The state-of-the-art aggregation models, however, either provide inconsistent predictions or require time-consuming aggregation process. We first prove the inconsistency of typical aggregations using disjoint or random data partition, and then present a consistent yet efficient aggregation model for large-scale GP. The proposed model inherits the advantages of aggregations, e.g., closed-form inference and aggregation, parallelization and distributed computing. Furthermore, theoretical and empirical analyses reveal that the new aggregation model performs better due to the consistent predictions that converge to the true underlying function when the training size approaches infinity.

Considering the driving habits which are learned from the naturalistic driving data in the path-tracking system can significantly improve the acceptance of intelligent vehicles. Therefore, the goal of this paper is to generate the prediction results of lateral commands with confidence regions according to the reference based on the learned motion primitives. We present a two-level structure for learning and generalizing motion primitives through demonstrations. The lower-level motion primitives are generated under the path segmentation and clustering layer in the upper-level. The Gaussian Mixture Model(GMM) is utilized to represent the primitives and Gaussian Mixture Regression (GMR) is selected to generalize the motion primitives. We show how the upper-level can help to improve the prediction accuracy and evaluate the influence of different time scales and the number of Gaussian components. The model is trained and validated by using the driving data collected from the Beijing Institute of Technology (BIT) intelligent vehicle platform. Experiment results show that the proposed method can extract the motion primitives from the driving data and predict the future lateral control commands with high accuracy.

Time series prediction has been studied in a variety of domains. However, it is still challenging to predict future series given historical observations and past exogenous data. Existing methods either fail to consider the interactions among different components of exogenous variables which may affect the prediction accuracy, or cannot model the correlations between exogenous data and target data. Besides, the inherent temporal dynamics of exogenous data are also related to the target series prediction, and thus should be considered as well. To address these issues, we propose an end-to-end deep learning model, i.e., Hierarchical attention-based Recurrent Highway Network (HRHN), which incorporates spatio-temporal feature extraction of exogenous variables and temporal dynamics modeling of target variables into a single framework. Moreover, by introducing the hierarchical attention mechanism, HRHN can adaptively select the relevant exogenous features in different semantic levels. We carry out comprehensive empirical evaluations with various methods over several datasets, and show that HRHN outperforms the state of the arts in time series prediction, especially in capturing sudden changes and sudden oscillations of time series.