The explosive growth of easily-accessible unlabeled data has lead to growing interest in active learning, a paradigm in which data-hungry learning algorithms adaptively select informative examples in order to lower prohibitively expensive labeling costs. Unfortunately, in standard worst-case models of learning, the active setting often provides no improvement over non-adaptive algorithms. To combat this, a series of recent works have considered a model in which the learner may ask enriched queries beyond labels. While such models have seen success in drastically lowering label costs, they tend to come at the expense of requiring large amounts of memory. In this work, we study what families of classifiers can be learned in bounded memory. To this end, we introduce a novel streaming-variant of enriched-query active learning along with a natural combinatorial parameter called lossless sample compression that is sufficient for learning not only with bounded memory, but in a query-optimal and computationally efficient manner as well. Finally, we give three fundamental examples of classifier families with small, easy to compute lossless compression schemes when given access to basic enriched queries: axis-aligned rectangles, decision trees, and halfspaces in two dimensions.

Clustering is a fundamental unsupervised learning task used for various applications, such as anomaly detection (Leung and Leckie, 2005), recommender systems (Shepitsen et al., 2008) and cancer diagnosis (Zheng et al., 2014). In recent years, research on sequential clustering has been actively studied, motivated by applications in which data arrives sequentially, such as online recommender systems (Nasraoui et al., 2007) and online community detection (Aggarwal, 2003). In this work, we study k-median clustering in the sequential no-substitution setting, a term first introduced in Hess and Sabato (2020). In this setting, a stream of data points is sequentially observed, and some of these points are selected by the algorithm as cluster centers. However, a point can be selected as a center only immediately after it is observed, before observing the next point. In addition, a selected center cannot be substituted later. This setting is motivated by applications in which center selection is mapped to a real-world irreversible action, such as providing users with promotional gifts or recruiting participants to a clinical trial. The goal in the no-substitution k-median setting is to obtain a near-optimal k-median risk value, while selecting a number of centers that is as close as possible to k.

Despite the popularity of explainable AI, there is limited work on effective methods for unsupervised learning. We study algorithms for $k$-means clustering, focusing on a trade-off between explainability and accuracy. Following prior work, we use a small decision tree to partition a dataset into $k$ clusters. This enables us to explain each cluster assignment by a short sequence of single-feature thresholds. While larger trees produce more accurate clusterings, they also require more complex explanations. To allow flexibility, we develop a new explainable $k$-means clustering algorithm, ExKMC, that takes an additional parameter $k' \geq k$ and outputs a decision tree with $k'$ leaves. We use a new surrogate cost to efficiently expand the tree and to label the leaves with one of $k$ clusters. We prove that as $k'$ increases, the surrogate cost is non-increasing, and hence, we trade explainability for accuracy. Empirically, we validate that ExKMC produces a low cost clustering, outperforming both standard decision tree methods and other algorithms for explainable clustering. Implementation of ExKMC available at https://github.com/navefr/ExKMC.

Clustering is a popular form of unsupervised learning for geometric data. Unfortunately, many clustering algorithms lead to cluster assignments that are hard to explain, partially because they depend on all the features of the data in a complicated way. To improve interpretability, we consider using a small decision tree to partition a data set into clusters, so that clusters can be characterized in a straightforward manner. We study this problem from a theoretical viewpoint, measuring cluster quality by the $k$-means and $k$-medians objectives: Must there exist a tree-induced clustering whose cost is comparable to that of the best unconstrained clustering, and if so, how can it be found? In terms of negative results, we show, first, that popular top-down decision tree algorithms may lead to clusterings with arbitrarily large cost, and second, that any tree-induced clustering must in general incur an $\Omega(\log k)$ approximation factor compared to the optimal clustering. On the positive side, we design an efficient algorithm that produces explainable clusters using a tree with $k$ leaves. For two means/medians, we show that a single threshold cut suffices to achieve a constant factor approximation, and we give nearly-matching lower bounds. For general $k \geq 2$, our algorithm is an $O(k)$ approximation to the optimal $k$-medians and an $O(k^2)$ approximation to the optimal $k$-means. Prior to our work, no algorithms were known with provable guarantees independent of dimension and input size.

In this paper we study k-means clustering in the online setting. In the offline setting the main parameters are number of centers, k, and size of the dataset, n. Performance guarantees are given as a function of these parameters. In the online setting new factors come into place: the ordering of the dataset and whether n is known in advance or not. One of the main results of this paper is the discovery that these new factors have dramatic effects on the quality of the clustering algorithms. For example, for constant k: (1) $\Omega(n)$ centers are needed if the order is arbitrary, (2) if the order is random and n is unknown in advance, the number of centers reduces to $\Theta(logn)$, and (3) if n is known, then the number of centers reduces to a constant. For different values of the new factors, we show upper and lower bounds that are exactly the same up to a constant, thus achieving optimal bounds.

Deep learning (DL) algorithms have successfully solved real-world classification problems from a variety of fields, including recognizing handwritten digits and identifying the presence of key diagnostic features in medical images [18, 16]. A typical classification challenge for a DL algorithm consists of training the algorithm on an example data set, then using a separate set of test data to evaluate its performance. The aim is to provide answers that are as accurate as possible, as measured by the true positive rate (TPR) and the true negative rate (TNR). Many DL classifiers, particularly those using a softmax function in the very last layer, yield a continuous score, h; A step function is used to map this continuous score to each of the possible categories that are being classified. TPR and TNR scores are then generated for each separate variable that is being predicted by setting a threshold parameter that is applied when mapping h to the decision. Values above this threshold are mapped to positive predictions, while values below it are mapped to negative predictions. The ROC curve is then generated from these pairs of TPR/TPN scores.