data structure
A Bayesian Nonparametric View on Count-Min Sketch
The count-min sketch is a time-and memory-efficient randomized data structure that provides a point estimate of the number of times an item has appeared in a data stream. The count-min sketch and related hash-based data structures are ubiquitous in systems that must track frequencies of data such as URLs, IP addresses, and language n-grams. We present a Bayesian view on the count-min sketch, using the same data structure, but providing a posterior distribution over the frequencies that characterizes the uncertainty arising from the hash-based approximation. In particular, we take a nonparametric approach and consider tokens generated from a Dirichlet process (DP) random measure, which allows for an unbounded number of unique tokens. Using properties of the DP, we show that it is possible to straightforwardly compute posterior marginals of the unknown true counts and that the modes of these marginals recover the count-min sketch estimator, inheriting the associated probabilistic guarantees. Using simulated data with known ground truth, we investigate the properties of these estimators. Lastly, we also study a modified problem in which the observation stream consists of collections of tokens (i.e., documents) arising from a random measure drawn from a stable beta process, which allows for power law scaling behavior in the number of unique tokens.
- North America > United States > Florida > Pinellas County > St. Petersburg (0.04)
- North America > United States > California > Santa Barbara County > Santa Barbara (0.04)
- North America > United States > Arizona > Maricopa County > Phoenix (0.04)
- (5 more...)
- Research Report > New Finding (0.46)
- Instructional Material > Course Syllabus & Notes (0.46)
- North America > United States > Massachusetts > Middlesex County > Cambridge (0.04)
- South America > Brazil > Rio de Janeiro > Rio de Janeiro (0.04)
- Research Report > Experimental Study (1.00)
- Research Report > New Finding (0.94)
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers
Despite several works trying to reduce their computational cost, most of LLMs still adopt attention layers between all pairs of tokens in the sequence, thus incurring a quadratic cost. In this study, we present a novel approach that dynamically prunes contextual information while preserving the model's expressiveness, resulting in reduced memory and computational
- Asia > Middle East > Lebanon (0.04)
- South America > Chile > Santiago Metropolitan Region > Santiago Province > Santiago (0.04)
- North America > United States > Oregon > Linn County > Lebanon (0.04)
- (7 more...)
- Research Report > Promising Solution (0.87)
- Research Report > New Finding (0.87)
- North America > United States > Pennsylvania > Allegheny County > Pittsburgh (0.14)
- North America > United States > California > Santa Clara County > Palo Alto (0.04)
- Europe > United Kingdom > England > Cambridgeshire > Cambridge (0.04)
- Africa > Eswatini > Manzini > Manzini (0.04)
- Africa > Senegal > Kolda Region > Kolda (0.05)
- North America > United States > District of Columbia > Washington (0.04)
- North America > Canada > Quebec > Montreal (0.04)
- Research Report (1.00)
- Workflow (0.68)
- Energy (0.93)
- Government > Regional Government > North America Government > United States Government (0.46)
- North America > United States > Virginia (0.04)
- South America > Brazil > Rio de Janeiro > Rio de Janeiro (0.04)
- North America > United States > Massachusetts (0.04)
- (5 more...)
- Information Technology > Artificial Intelligence > Machine Learning > Statistical Learning (0.93)
- Information Technology > Artificial Intelligence > Natural Language (0.93)
- Information Technology > Artificial Intelligence > Representation & Reasoning > Mathematical & Statistical Methods (0.67)
- Information Technology > Artificial Intelligence > Machine Learning > Neural Networks > Deep Learning (0.46)