Transformer models have achieved state-of-the-art results across a diverse range of domains. However, concern over the cost of training the attention mechanism to learn complex dependencies between distant inputs continues to grow. In response, solutions that exploit the structure and sparsity of the learned attention matrix have blossomed. However, real-world applications that involve long sequences, such as biological sequence analysis, may fall short of meeting these assumptions, precluding exploration of these models. To address this challenge, we present a new Transformer architecture, Performer, based on Fast Attention Via Orthogonal Random features (FAVOR). Our mechanism scales linearly rather than quadratically in the number of tokens in the sequence, is characterized by sub-quadratic space complexity and does not incorporate any sparsity pattern priors. Furthermore, it provides strong theoretical guarantees: unbiased estimation of the attention matrix and uniform convergence. It is also backwards-compatible with pre-trained regular Transformers. We demonstrate its effectiveness on the challenging task of protein sequence modeling and provide detailed theoretical analysis.

Pretrained language models, especially masked language models (MLMs) have seen success across many NLP tasks. However, there is ample evidence that they use the cultural biases that are undoubtedly present in the corpora they are trained on, implicitly creating harm with biased representations. To measure some forms of social bias in language models against protected demographic groups in the US, we introduce the Crowdsourced Stereotype Pairs benchmark (CrowS-Pairs). CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pairs a model is presented with two sentences: one that is more stereotyping and another that is less stereotyping. The data focuses on stereotypes about historically disadvantaged groups and contrasts them with advantaged groups. We find that all three of the widely-used MLMs we evaluate substantially favor sentences that express stereotypes in every category in CrowS-Pairs. As work on building less biased models advances, this dataset can be used as a benchmark to evaluate progress.

The region expects to collect almost 15% of its GDP – roughly $95b – by 2030 from its growing AI industry. Toward this end, it has invested heavily in the sector. To actualize the goal, UAE created the position of'Minister of State for AI' in 2017 to boost citizens' participation in the technological drive. This is a clear indication that the government wants to hold a firm grip on disruptive technology. Omar bin Sultan Al Olama, the current Minister of State for AI, recently advocated for the technology in an interview.

A new artificial intelligence tool could be the key to saving lives and reducing conflict between people and the decreasing populations of endangered Asian and African elephants. African elephant populations have dropped from 12 million to 400,000 in the past century, according to the World Wide Fund For Nature (WWF). There are also now fewer than 50,000 Asian elephants left in existence, according to WWF estimates. The gentle mammals face an onslaught of threats, from the illegal ivory trade to deforestation, which has forced the elephants to expand into human-inhabited areas and has increased conflict with people. Farmers worldwide -- including in India, Thailand, and Africa -- have frequently reported negative interactions with elephants grazing on crops or entering villages.

Echoing the voice of all tech innovators and government agencies, global leaders at Taiwan have pitched AI ML solutions for COVID-19 and the contactless economy. Today, our lives are controlled by promising innovations to drive the contactless economy. The global economy has slowed down and shrunk significantly in the last six months due to lockdown protocols. The COVID-19 pandemic has been the single biggest catastrophe to have derailed the global scenario. While we were aware of mostly contactless payments and digital messaging solutions making it big in the Pre-COVID-19 days, today's scenario is totally different.

Orchids are more than just decorative - they are also economically important in horticulture, in the pharmaceutical industry and even in the food industry. For example, vanilla orchids are grown commercially for their seed pods, and the economy on the northeast of Madagascar centers around the vanilla trade. But many of the approximately 29,000 orchid species face immediate threats by land conversion and illegal harvesting, resulting in an urgent need to identify the most endangered species and protect them from extinction. The global Red List of the International Union for the Conservation of Nature (IUCN) is the most widely used scheme to evaluate species' risk of extinction. The assessments are based on rigorous criteria and the best available scientific information, making them resource-intensive and, therefore, only available for a fraction of the species worldwide.

Advances on signal, image and video generation underly major breakthroughs on generative medical imaging tasks, including Brain Image Synthesis. Still, the extent to which functional Magnetic Ressonance Imaging (fMRI) can be mapped from the brain electrophysiology remains largely unexplored. This work provides the first comprehensive view on how to use state-of-the-art principles from Neural Processing to synthesize fMRI data from electroencephalographic (EEG) data. Given the distinct spatiotemporal nature of haemodynamic and electrophysiological signals, this problem is formulated as the task of learning a mapping function between multivariate time series with highly dissimilar structures. A comparison of state-of-the-art synthesis approaches, including Autoencoders, Generative Adversarial Networks and Pairwise Learning, is undertaken. Results highlight the feasibility of EEG to fMRI brain image mappings, pinpointing the role of current advances in Machine Learning and showing the relevance of upcoming contributions to further improve performance. EEG to fMRI synthesis offers a way to enhance and augment brain image data, and guarantee access to more affordable, portable and long-lasting protocols of brain activity monitoring. The code used in this manuscript is available in Github and the datasets are open source.

How can we effectively regularize BERT? Although BERT proves its effectiveness in various downstream natural language processing tasks, it often overfits when there are only a small number of training instances. A promising direction to regularize BERT is based on pruning its attention heads based on a proxy score for head importance. However, heuristic-based methods are usually suboptimal since they predetermine the order by which attention heads are pruned. In order to overcome such a limitation, we propose AUBER, an effective regularization method that leverages reinforcement learning to automatically prune attention heads from BERT. Instead of depending on heuristics or rule-based policies, AUBER learns a pruning policy that determines which attention heads should or should not be pruned for regularization. Experimental results show that AUBER outperforms existing pruning methods by achieving up to 10% better accuracy. In addition, our ablation study empirically demonstrates the effectiveness of our design choices for AUBER.

Looked at on a world scale, the Covid-19 pandemic will continue to deliver shocks to global supply chains for some time to come. Even if the public health crisis abates in the UK, our economy is part of a global economy, and UK corporate IT will have its work cut out in supporting companies as they are forced to re-forge supply chains, perhaps over and over again, and at short notice. The crisis has provoked some rethinking of how the world economy ought to work, with an emphasis on the desirability of a shift from efficiency – doing things "just in time" – to resilience – building in more slack. The FT's Rana Faroohar provides an account of such rethinking in an article entitled From'just in time' to'just in case' published earlier this year. In the discussions which lie behind this article there are different emphases on a spectrum of opinion: some say we can have both efficiency and resilience equally, others that there is a choice to be made for one or the other, and yet others say it's a matter of balance, of trading off. Tony Harris, global vice-president of business network solutions at SAP, says it has to be a combination. "You wouldn't want to move to a resilient network or supply chain that wasn't also efficient," he says.

Artificial intelligence and machine learning are fast moving out of the hype status and into the reality status. South African industry recently launched the AI Institute of South Africa, with the objective of consulting with government on AI policy, while also representing South Africa in global AI forums and initiatives. The AI world is faced with many questions. How can we ensure that AI is for good and will not destroy itself and in the process, the world? Do we need global regulations?