If you would like to read about DistilBERT in more detail I'd suggest clicking here for the article, but from what the abstract suggests it was made 60% faster by performing a 40% size reduction while retaining 97% of its language understanding. This is a significant improvement and a great optimization with respect to traditional or'vanilla' BERT. As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.

Deep learning, a subset of machine learning based on artificial neural networks, has enabled applications with performance levels approaching those of …

The ability of computers to understand and generate language took a huge leap forward in 2017 when researchers at Google developed new natural -anguage AI models called Transformers. Some of the experts who built and trained those seminal models have since branched out on their own by founding the Toronto-based startup Cohere, which today announced a new $40 million Series A funding round. The technology that undergirds Cohere's natural-language processing models was originally developed by the Toronto-based Google Brain team. Two of that team's members, Aidan Gomez and Nick Frosst (along with a third cofounder, Ivan Zhang), started Cohere two years ago to further develop and commercialize the models, which are delivered to customers through an API. Cohere is backed by neural network pioneer and Turing Award winner Geoffrey Hinton, who led the Toronto Google Brain team, as well as some other big names in the AI world like Stanford computer science professor Fei-Fei Li. "Very large language models are now giving computers a much better understanding of human communication," Hinton said in a statement to Fast Company.

Next up is adding artificial intelligence to your skillset with An Easy Introduction To AI And Deep Learning, which will cover self-driving cars, Siri …

Natural language processing (NLP) is a subfield of AI and linguistics which enables computers to understand, interpret and manipulate human language. Although NLP faces different challenges due to the difficulty of human language, this did not become an obstacle in the face of its growth. The global NLP market was estimated at $5B in 2018 and is expected to reach $43B by 2025, and this exponential growth can mostly be attributed to the vast use cases of NLP in every industry today. You may already be familiar with many NLP applications such as autocorrection, translation, or chatbots. However, NLP is the cornerstone of numerous applications we use every day without even noticing.

In their paper, Tabular Data: Deep Learning is Not All You Need, the authors argue that while deep learning methods have shown tremendous success in the image and text domains, traditional tree-based methods like XGBoost still continue to shine when it comes to tabular data. The authors examined Tabnet, Neural Oblivious Decision Ensembles (NODE), DNF-Net, and 1D-CNN deep learning models and compared their performance on eleven datasets with XGBoost. This is an important paper in the sense that it reiterates that deep learning may not be the silver bullet for solving all machine learning problems. On the other hand, tree-based algorithms have been shown to perform at par or even outperform neural networks for tabular data while being simple to use and comprehend. And there is good news for people who like to work with tree-based models.

This work presents a sparse-attention Transformer architecture for modeling documents that contain large tables. Tables are ubiquitous on the web, and are rich in information. However, more than 20% of relational tables on the web have 20 or more rows (Cafarella et al., 2008), and these large tables present a challenge for current Transformer models, which are typically limited to 512 tokens. Here we propose MATE, a novel Transformer architecture designed to model the structure of web tables. MATE uses sparse attention in a way that allows heads to efficiently attend to either rows or columns in a table. This architecture scales linearly with respect to speed and memory, and can handle documents containing more than 8000 tokens with current accelerators. MATE also has a more appropriate inductive bias for tabular data, and sets a new state-of-the-art for three table reasoning datasets. For HybridQA (Chen et al., 2020b), a dataset that involves large documents containing tables, we improve the best prior result by 19 points.

A common problem in the task of human-object interaction (HOI) detection is that numerous HOI classes have only a small number of labeled examples, resulting in training sets with a long-tailed distribution. The lack of positive labels can lead to low classification accuracy for these classes. Towards addressing this issue, we observe that there exist natural correlations and anti-correlations among human-object interactions. In this paper, we model the correlations as action co-occurrence matrices and present techniques to learn these priors and leverage them for more effective training, especially on rare classes. The efficacy of our approach is demonstrated experimentally, where the performance of our approach consistently improves over the state-of-the-art methods on both of the two leading HOI detection benchmark datasets, HICO-Det and V-COCO.

Electroencephalogram (EEG) has shown a useful approach to produce a brain-computer interface (BCI). One-dimensional (1-D) EEG signal is yet easily disturbed by certain artifacts (a.k.a. noise) due to the high temporal resolution. Thus, it is crucial to remove the noise in received EEG signal. Recently, deep learning-based EEG signal denoising approaches have achieved impressive performance compared with traditional ones. It is well known that the characteristics of self-similarity (including non-local and local ones) of data (e.g., natural images and time-domain signals) are widely leveraged for denoising. However, existing deep learning-based EEG signal denoising methods ignore either the non-local self-similarity (e.g., 1-D convolutional neural network) or local one (e.g., fully connected network and recurrent neural network). To address this issue, we propose a novel 1-D EEG signal denoising network with 2-D transformer, namely EEGDnet. Specifically, we comprehensively take into account the non-local and local self-similarity of EEG signal through the transformer module. By fusing non-local self-similarity in self-attention blocks and local self-similarity in feed forward blocks, the negative impact caused by noises and outliers can be reduced significantly. Extensive experiments show that, compared with other state-of-the-art models, EEGDnet achieves much better performance in terms of both quantitative and qualitative metrics.