Libya
Country:
- Europe > Ukraine > Kyiv Oblast > Kyiv (0.14)
- Europe > Austria > Vienna (0.14)
- Asia > Japan > Honshū > Kantō > Tokyo Metropolis Prefecture > Tokyo (0.14)
- (96 more...)
Genre:
- Research Report > New Finding (1.00)
- Research Report > Experimental Study (1.00)
Industry:
- Education > Health & Safety > School Nutrition (0.93)
- Health & Medicine > Consumer Health (0.93)
Technology:
- Information Technology > Artificial Intelligence > Representation & Reasoning (1.00)
- Information Technology > Artificial Intelligence > Natural Language > Large Language Model (1.00)
- Information Technology > Artificial Intelligence > Natural Language > Chatbot (0.73)
- Information Technology > Artificial Intelligence > Machine Learning > Neural Networks > Deep Learning (0.52)
Language Model Tokenizers Introduce Unfairness Between Languages
Recent language models have shown impressive multilingual performance, even when not explicitly trained for it. Despite this, there are concerns about the quality of their outputs across different languages. In this paper, we show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked. The same text translated into different languages can have drastically different tok-enization lengths, with differences up to 15 times in some cases. These disparities persist even for tokenizers that are intentionally trained for multilingual support.
Country:
- North America > Haiti (0.14)
- Asia > Philippines > Luzon > Ilocos Region > Province of Pangasinan (0.04)
- Europe > Switzerland > Zürich > Zürich (0.04)
- (38 more...)
Technology:
- Information Technology > Artificial Intelligence > Natural Language > Large Language Model (0.70)
- Information Technology > Artificial Intelligence > Natural Language > Chatbot (0.69)
- Information Technology > Artificial Intelligence > Machine Learning > Neural Networks > Deep Learning (0.69)
- Information Technology > Artificial Intelligence > Natural Language > Machine Translation (0.68)
Country:
- North America > United States > California > San Francisco County > San Francisco (0.14)
- North America > United States > Virginia (0.04)
- North America > United States > Texas (0.04)
- (10 more...)
Technology:
- Information Technology > Communications > Social Media (1.00)
- Information Technology > Artificial Intelligence > Representation & Reasoning (1.00)
- Information Technology > Artificial Intelligence > Machine Learning > Statistical Learning > Clustering (1.00)
- Information Technology > Data Science > Data Mining (0.88)
Country:
- Europe > United Kingdom > England > Oxfordshire > Oxford (0.14)
- Europe > Austria > Vienna (0.14)
- Oceania (0.05)
- (29 more...)
Industry:
- Law (1.00)
- Government (0.93)
Technology:
Country:
- North America > United States > New York (0.04)
- North America > United States > Colorado (0.04)
- North America > United States > Arizona > Pima County (0.04)
- (2 more...)
Industry:
- Media > News (1.00)
- Government > Regional Government > North America Government > United States Government (0.70)
Country:
- North America > United States > Colorado (0.04)
- North America > United States > Arizona > Pima County (0.04)
- Europe > Spain (0.04)
- Africa > Middle East > Libya > Benghazi District > Benghazi (0.04)
Industry:
- Media > News (1.00)
- Information Technology > Security & Privacy (1.00)
- Government > Regional Government > North America Government > United States Government (0.69)
- Health & Medicine > Therapeutic Area > Psychiatry/Psychology (0.48)
Technology:
Country:
- North America > United States > Virginia (0.05)
- Asia > Taiwan (0.05)
- North America > United States > Illinois > Cook County > Chicago (0.04)
- (2 more...)
Industry:
- Media > News (1.00)
- Government > Regional Government > North America Government > United States Government (0.71)
- Health & Medicine > Therapeutic Area > Psychiatry/Psychology (0.49)
Technology:
Country:
- North America > United States > Minnesota (0.05)
- North America > United States > Texas (0.04)
- North America > United States > District of Columbia > Washington (0.04)
- (4 more...)
Industry:
- Media > News (1.00)
- Leisure & Entertainment > Sports > Football (1.00)
- Government > Regional Government > North America Government > United States Government (1.00)
Technology:
- Information Technology > Communications > Social Media (1.00)
- Information Technology > Artificial Intelligence > Natural Language > Chatbot (0.34)
Country:
- North America > United States > California > Santa Clara County > Palo Alto (0.05)
- North America > United States > Washington (0.04)
- North America > United States > Arizona > Pima County (0.04)
- (2 more...)
Industry:
- Media > News (1.00)
- Government > Regional Government > North America Government > United States Government (1.00)
Technology:
Country:
- North America > United States > Texas > Dallas County > Dallas (0.04)
- North America > United States > Arizona > Pima County (0.04)
- Europe > Spain (0.04)
- Africa > Middle East > Libya > Benghazi District > Benghazi (0.04)
Industry:
- Media > News (1.00)
- Government > Regional Government > North America Government > United States Government (0.70)
- Health & Medicine > Therapeutic Area > Psychiatry/Psychology (0.50)
Technology: