Goto

Collaborating Authors

Top Emerging Deep Learning Trends For 2022

#artificialintelligence

In today's industry, AI and machine learning are regarded as the cornerstones of technological transformation. Enterprises have become more intelligent and efficient as a result of incorporating machine learning algorithms into their operations. The advancement of deep learning has gained the attention of industry experts and IT companies as the next paradigm shift in computing is underway. Deep learning technology is now widely used in a variety of businesses around the world. The deep learning revolution is centered on artificial neural networks. Experts believe that the advent of machine learning and related technologies has reduced overall error rates and increased the effectiveness of networks for specific tasks.


A Survey on Visual Transformer

arXiv.org Artificial Intelligence

Transformer is a type of deep neural network mainly based on self-attention mechanism which is originally applied in natural language processing field. Inspired by the strong representation ability of transformer, researchers propose to extend transformer for computer vision tasks. Transformer-based models show competitive and even better performance on various visual benchmarks compared to other network types such as convolutional networks and recurrent networks. With high performance and without inductive bias defined by human, transformer is receiving more and more attention from the visual community. In this paper we provide a literature review of these visual transformer models by categorizing them in different tasks and analyze the advantages and disadvantages of these methods. In particular, the main categories include the basic image classification, high-level vision, low-level vision and video processing. The self-attention in computer vision is also briefly revisited as self-attention is the base component in transformer. Efficient transformer methods are included for pushing transformer into real applications on the devices. Finally, we give a discussion about the challenges and further research directions for visual transformers.


Transformers in Vision: A Survey

arXiv.org Artificial Intelligence

Astounding results from transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. This has led to exciting progress on a number of tasks while requiring minimal inductive biases in the model design. This survey aims to provide a comprehensive overview of the transformer models in the computer vision discipline and assumes little to no prior background in the field. We start with an introduction to fundamental concepts behind the success of transformer models i.e., self-supervision and self-attention. Transformer architectures leverage self-attention mechanisms to encode long-range dependencies in the input domain which makes them highly expressive. Since they assume minimal prior knowledge about the structure of the problem, self-supervision using pretext tasks is applied to pre-train transformer models on large-scale (unlabelled) datasets. The learned representations are then fine-tuned on the downstream tasks, typically leading to excellent performance due to the generalization and expressivity of encoded features. We cover extensive applications of transformers in vision including popular recognition tasks (e.g., image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks (e.g., visual-question answering and visual reasoning), video processing (e.g., activity recognition, video forecasting), low-level vision (e.g., image super-resolution and colorization) and 3D analysis (e.g., point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works.


The Rise of the Transformers

#artificialintelligence

Rise of the Transformers with Self-Attention Mechanism  The intention of this article is to continue in answering the questions that my friends April Rudin, Tripp Braden, Danielle Guzman and Richard Foster-Fletcher asked about the future of AI. Furthermore Irene Iyakovet interview with me about how


MalBERT: Using Transformers for Cybersecurity and Malicious Software Detection

arXiv.org Artificial Intelligence

In recent years we have witnessed an increase in cyber threats and malicious software attacks on different platforms with important consequences to persons and businesses. It has become critical to find automated machine learning techniques to proactively defend against malware. Transformers, a category of attention-based deep learning techniques, have recently shown impressive results in solving different tasks mainly related to the field of Natural Language Processing (NLP). In this paper, we propose the use of a Transformers' architecture to automatically detect malicious software. We propose a model based on BERT (Bidirectional Encoder Representations from Transformers) which performs a static analysis on the source code of Android applications using preprocessed features to characterize existing malware and classify it into different representative malware categories. The obtained results are promising and show the high performance obtained by Transformer-based models for malicious software detection.