This article explores deep learning models for the automatic identification of registers - text varieties such as news reports and discussion forums - in web-based datasets across 16 languages. Web register (or genre) identification would provide a robust solution for understanding the content of web-scale datasets, which have become crucial in computational linguistics. Despite recent advances, the potential of register classifiers on the noisy web remains largely unexplored, particularly in multilingual settings and when targeting the entire unrestricted web. We experiment with a range of deep learning models using the new Multilingual CORE corpora, which includes 16 languages annotated using a detailed, hierarchical taxonomy of 25 registers designed to cover the entire unrestricted web. Our models achieve state-of-the-art results, showing that a detailed taxonomy in a hierarchical multi-label setting can yield competitive classification performance. However, all models hit a glass ceiling at approximately 80% F1 score, which we attribute to the non-discrete nature of web registers and the inherent uncertainty in labeling some documents. By pruning ambiguous examples, we improve model performance to over 90%. Finally, multilingual models outperform monolingual ones, particularly benefiting languages with fewer training examples and smaller registers. Although a zero-shot setting decreases performance by an average of 7%, these drops are not linked to specific registers or languages. Instead, registers show surprising similarity across languages.

Gain actionable insights from peers and AI experts during a live Twitter chat. On Oct. 15, 2019, MIT Sloan Management Review and Boston Consulting Group published their most recent collaborative research report examining the corporate adoption of and impact from artificial intelligence. We invite you to join us on Twitter for an engaging discussion of some of the themes we address in the report. Please join report writer Julia Kirby (@juliakirby) and Martin Reeves (@martinkreeves), global director of the BCG Henderson Institute, Wednesday, Oct. 23, 2019, at 11 a.m. PDT, to join what promises to be a fast-paced and informative conversation.