Stanford AI researchers make 'socially inclusive' NLP using Urban Dictionary and Twitter

Stanford University AI researchers have created a "socially equitable" natural language processing (NLP) tool they say improves upon off-the-shelf AI solutions used today that fail to account for things like regional dialects, slang, or the natural way people talk when they regularly speak more than one language. In a paper published late last week, researchers found Equilid to be more accurate than commonly used identification tools like langid.py and Google's CLD2. Popular language identification tools, the paper argues, draw on a "European-centric corpora" of the written word, as well as websites, Wikipedia, and newswires, methods that may not best represent the way people actually talk. Language identification is a form of NLP used for things like serving up Google search results or even tracking social media chatter to make predictions. Equilid was made to better understand slang, regional dialects, and the natural way people communicate online when they speak more than one language, like, say, the 90 million English speakers in the Philippines who may regularly switch between English and Tagalog.