The vast majority of the popular English named entity recognition (NER) datasets contain American or British English data, despite the existence of many global varieties of English. As such, it is unclear whether they generalize for analyzing use of English globally. To test this, we build a newswire dataset, the Worldwide English NER Dataset, to analyze NER model performance on low-resource English variants from around the world. We test widely used NER toolkits and transformer models, including models using the pre-trained contextual models RoBERTa and ELECTRA, on three datasets: a commonly used British English newswire dataset, CoNLL 2003, a more American focused dataset OntoNotes, and our global dataset. All models trained on the CoNLL or OntoNotes datasets experienced significant performance drops-over 10 F1 in some cases-when tested on the Worldwide English dataset. Upon examination of region-specific errors, we observe the greatest performance drops for Oceania and Africa, while Asia and the Middle East had comparatively strong performance. Lastly, we find that a combined model trained on the Worldwide dataset and either CoNLL or OntoNotes lost only 1-2 F1 on both test sets.

A machine learning model was used by researchers from the University of Waikato, in New Zealand, to narrow down a massive 8 million tweets to a more manageable 1.2 million in order to look at how te reo Māori is being used in the genre. According to a recent press release, the team focused on 77 Māori loanwords, or te reo Māori words used in an English context, and used them as training data for their machine learning model. Machine learning allows data scientists to provide a computer with a large data set, and teach it to make predictions based on that data. The initial 8 million tweets contained a fair bit of distracting data'noise'. The irrelevant tweets are those that are not used in a New Zealand English context, or were otherwise unrelated.

Researchers have whittled down a massive 8 million tweets, to a more manageable 1.2 million to look at how te reo MÄ ori is being used in the genre. The team from the University of Waikato have focused on 77 MÄ ori loanwords (te reo MÄ ori words used in an English context) and used them as training data for their machine-learning model. Machine learning allows data scientists to provide a computer with a large data set, and teach it to make predictions based on that data. Computing and Mathematical Sciences student David Trye spent the summer working on the project, with supervisorsDr Andreea Calude and Dr Felipe Bravo Márquez. The initial 8-million tweets contained a fair bit of distracting data'noise'.