Random Fourier Features (RFF) is among the most popular and broadly applicable approaches for scaling up kernel methods. In essence, RFF allows the user to avoid costly computations on a large kernel matrix via a fast randomized approximation. However, a pervasive difficulty in applying RFF is that the user does not know the actual error of the approximation, or how this error will propagate into downstream learning tasks. Up to now, the RFF literature has primarily dealt with these uncertainties using theoretical error bounds, but from a user's standpoint, such results are typically impractical -- either because they are highly conservative or involve unknown quantities. To tackle these general issues in a data-driven way, this paper develops a bootstrap approach to numerically estimate the errors of RFF approximations. Three key advantages of this approach are: (1) The error estimates are specific to the problem at hand, avoiding the pessimism of worst-case bounds. (2) The approach is flexible with respect to different uses of RFF, and can even estimate errors in downstream learning tasks. (3) The approach enables adaptive computation, so that the user can quickly inspect the error of a rough initial kernel approximation and then predict how much extra work is needed. Lastly, in exchange for all of these benefits, the error estimates can be obtained at a modest computational cost.

In this work, we present a novel global descriptor termed stable triangle descriptor (STD) for 3D place recognition. For a triangle, its shape is uniquely determined by the length of the sides or included angles. Moreover, the shape of triangles is completely invariant to rigid transformations. Based on this property, we first design an algorithm to efficiently extract local key points from the 3D point cloud and encode these key points into triangular descriptors. Then, place recognition is achieved by matching the side lengths (and some other information) of the descriptors between point clouds. The point correspondence obtained from the descriptor matching pair can be further used in geometric verification, which greatly improves the accuracy of place recognition. In our experiments, we extensively compare our proposed system against other state-of-the-art systems (i.e., M2DP, Scan Context) on public datasets (i.e., KITTI, NCLT, and Complex-Urban) and our self-collected dataset (with a non-repetitive scanning solid-state LiDAR). All the quantitative results show that STD has stronger adaptability and a great improvement in precision over its counterparts. To share our findings and make contributions to the community, we open source our code on our GitHub: https://github.com/hku-mars/STD.

Recent years have witnessed increasing concerns towards unfair decisions made by machine learning algorithms. To improve fairness in model decisions, various fairness notions have been proposed and many fairness-aware methods are developed. However, most of existing definitions and methods focus only on single-label classification. Fairness for multi-label classification, where each instance is associated with more than one labels, is still yet to establish. To fill this gap, we study fairness-aware multi-label classification in this paper. We start by extending Demographic Parity (DP) and Equalized Opportunity (EOp), two popular fairness notions, to multi-label classification scenarios. Through a systematic study, we show that on multi-label data, because of unevenly distributed labels, EOp usually fails to construct a reliable estimate on labels with few instances. We then propose a new framework named Similarity $s$-induced Fairness ($s_\gamma$-SimFair). This new framework utilizes data that have similar labels when estimating fairness on a particular label group for better stability, and can unify DP and EOp. Theoretical analysis and experimental results on real-world datasets together demonstrate the advantage of over existing methods $s_\gamma$-SimFair on multi-label classification tasks.

In this big data era, it is hard for the current generation to find the right data from the huge amount of data contained within online platforms. In such a situation, there is a need for an information filtering system that might help them find the information they are looking for. In recent years, a research field has emerged known as recommender systems. Recommenders have become important as they have many real-life applications. This paper reviews the different techniques and developments of recommender systems in e-commerce, e-tourism, e-resources, e-government, e-learning, and e-library. By analyzing recent work on this topic, we will be able to provide a detailed overview of current developments and identify existing difficulties in recommendation systems. The final results give practitioners and researchers the necessary guidance and insights into the recommendation system and its application.

Evaluation studies frequently rely on simplified experimental settings with non-expert users (e.g., workers on Amazon Mechanical Turk), use proxy tasks (e.g., forward simulation), or use subjective, user-reported measures as metrics of explanation quality [9, 16, 18, 19, 25, 26, 31]. Such settings are not equipped to evaluate the real-world utility of explainable ML methods since proxy task performance does not reflect real-task performance [3], users' perception of explanation usefulness is not reflective of utility in a task [3, 17], and proxy users do not reflect how expert users would use explanations [1]. A few studies evaluate explainable ML methods on their intended deployment settings where domain expert users perform the intended task [10, 20] (dubbed application-grounded evaluation studies in [6]). However, even in those, we argue that experimental design flaws (e.g., not isolating the incremental impact of explanations in [20]) and seemingly trivial design choices that deviate experimental settings from the deployment context (e.g., using metrics that do not reflect the task objectives in [10]), limit the applicability of drawn conclusions. We elaborate on these limitations in Section 2. In this work, we seek to bridge this critical gap by conducting a study that evaluates explainable ML methods in a setting consistent with the intended deployment context. Our study builds on the e-commerce fraud detection setting used in a previous evaluation study [10] consisting of professional fraud analysts tasked with reviewing e-commerce transactions to detect fraud when the ML model is uncertain about the outcome. We identify several simplifying assumptions made by the previous study that deviated from the deployment context and modify the setup to relax those assumptions (summarized in Table 1 and discussed in detail in Section 3.2). These modifications make the experimental setup faithful to the deployment setting and equipped to evaluate the utility of the explainable ML methods considered. Our setup results in dramatically different conclusions of the relative utility of ML model scores and explanations compared to the earlier work [10].

Binary classification is an important problem in many areas such as computer vision, natural language processing, and bioinformatics. Binary classification learning algorithms result in a function that outputs a real-valued predicted score (larger for more likely to be in the positive class). The prediction accuracy of learned binary classification models can be quantified using the zero-one loss, which corresponds to thresholding the predicted score at zero. Because it only considers one prediction threshold (the default), this evaluation metric can be problematic and/or misleading in some cases (data sets with extreme class imbalance, models with different false positive rates). A more comprehensive and fair evaluation method involves the Receiver Operating Characteristic (ROC) Curve, which involves plotting True Positive Rate versus False Positive Rate, for all thresholds of the predicted score [Egan and Egan, 1975]. The Area Under the ROC Curve (AUC) takes values between zero and one; constant/random/un-informed predictions yield AUC=0.5 and a set of perfect predictions would achieve AUC=1. It is therefore desirable to create learning algorithms that maximize AUC, and that criterion is often used for hyper-parameter selection. However, for gradient descent learning it is impossible to directly use the AUC since it is a piecewise constant function of the predicted values (the gradient is zero almost everywhere). Various authors have proposed to work around this issue by using convex relaxations of the Mann-Whitney statistic [Bamber, 1975], which involves a double sum over all pairs of positive and negative examples.

The parameters of a machine learning model are typically learned by minimizing a loss function on a set of training data. However, this can come with the risk of overtraining; in order for the model to generalize well, it is of great importance that we are able to find the optimal parameter for the model on the entire population -- not only on the given training sample. In this paper, we construct valid confidence sets for this optimal parameter of a machine learning model, which can be generated using only the training data without any knowledge of the population. We then show that studying the distribution of this confidence set allows us to assign a notion of confidence to arbitrary regions of the parameter space, and we demonstrate that this distribution can be well-approximated using bootstrapping techniques.

The current certification process for aerospace software is not adapted to "AI-based" algorithms such as deep neural networks. Unlike traditional aerospace software, the precise parameters optimized during neural network training are as important as (or more than) the code processing the network and they are not directly mathematically understandable. Despite their lack of explainability such algorithms are appealing because for some applications they can exhibit high performance unattainable with any traditional explicit line-by-line software methods. This paper proposes a framework and principles that could be used to establish certification methods for neural network models for which the current certification processes such as DO-178 cannot be applied. While it is not a magic recipe, it is a set of common sense steps that will allow the applicant and the regulator increase their confidence in the developed software, by demonstrating the capabilities to bring together, trace, and track the requirements, data, software, training process, and test results.

Epilepsy is a chronic neurological disorder that affects a significant portion of the human population and imposes serious risks in the daily life of patients. Despite advances in machine learning and IoT, small, nonstigmatizing wearable devices for continuous monitoring and detection in outpatient environments are not yet available. Part of the reason is the complexity of epilepsy itself, including highly imbalanced data, multimodal nature, and very subject-specific signatures. However, another problem is the heterogeneity of methodological approaches in research, leading to slower progress, difficulty comparing results, and low reproducibility. Therefore, this article identifies a wide range of methodological decisions that must be made and reported when training and evaluating the performance of epilepsy detection systems. We characterize the influence of individual choices using a typical ensemble random-forest model and the publicly available CHB-MIT database, providing a broader picture of each decision and giving good-practice recommendations, based on our experience, where possible.

How can one meaningfully make a measurement, if the meter does not conform to any standard and its scale expands or shrinks depending on what is measured? In the present work it is argued that current evaluation practices for machine-learning classifiers are affected by this kind of problem, leading to negative consequences when classifiers are put to real use; consequences that could have been avoided. It is proposed that evaluation be grounded on Decision Theory, and the implications of such foundation are explored. The main result is that every evaluation metric must be a linear combination of confusion-matrix elements, with coefficients - "utilities" - that depend on the specific classification problem. For binary classification, the space of such possible metrics is effectively two-dimensional. It is shown that popular metrics such as precision, balanced accuracy, Matthews Correlation Coefficient, Fowlkes-Mallows index, F1-measure, and Area Under the Curve are never optimal: they always give rise to an in-principle avoidable fraction of incorrect evaluations. This fraction is even larger than would be caused by the use of a decision-theoretic metric with moderately wrong coefficients.