My PhD student Erwei Wang, various collaborators and I have recently published a detailed survey article on this topic: Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going, to appear in ACM CSUR. In this post, I will informally explain my personal view of the role of approximation in supervised learning (classification), and how this links to the very active topic of DNN accelerator design in hardware. We can think of a DNN as a graph, where nodes perform computations and edges carry data. This graph can be interpreted (executed) as a function mapping input data to output data. The quality of this DNN is typically judged by a loss function .

We will start with a quick review then discuss overfitting, data augmentation, and end with a discussion on training validation development and testing. In classical machine learning, traditionally you have an image that is end by end, and you engineer a smaller set of K features. These features can be for example the ratio of the length to the height of the object, or the number of circle objects in the image. Then you apply your valid algorithm to learn to associate these patterns of features with an identity, In this case, vehicle. With supervised learning, we should learn algorithms used pre-labled training data to infer a function. Here imagine that you're trying to classify image into five categories. Vehicles, animals, faces, fruits, and chairs using two features, the two axises here. Supervised learning using the mission learning algorithms listed here allows for the determination of decision boundaries, shown here, using pre-labeled data. Here we can see that, while the decision boundaries are not perfect, they do a reasonable good job classifying the data.

A few days ago DeepMind announced their latest iteration of their StarCraft II machine learning bot, now known as AlphaStar. So much has changed since they announced the StarCraft II Learning Environment back in August 2017. David Silver, highly regarded for his reinforcement learning experience with projects like AlphaGo, has joined the team, which seems to have grown considerably. Back at Blizzcon in 2018, DeepMind demonstrated a bot that achieved a win rate of around 50% against the hardest game AI, pretty good but not mind blowing, so what changed in the last few months? I imagine that they would love to have demonstrated the decisive defeat of TLO and MaNa, but it seems they weren't ready.

There is a common saying, "A picture is worth a thousand words". In this post, we are going to take that literally and try to find the words in a picture! In an earlier post about Text Recognition, we discussed how Tesseract works and how it can be used along with OpenCV for text detection as well as recognition. This time, we are going to have a look at robust approach for detecting text, based on a recent paper: EAST: An Efficient and Accurate Scene Text Detector. It should be noted that text detection is different from text recognition.

Language understanding is a challenge for computers. Subtle nuances of communication that human toddlers can understand still confuse the most powerful machines. Even though advanced techniques like deep learning can detect and replicate complex language patterns, machine learning models still lack fundamental conceptual understanding of what our words really mean. That said, 2018 did yield a number of landmark research breakthroughs which pushed the fields of natural language processing, understanding, and generation forward. We summarized 14 research papers covering several advances in natural language processing (NLP), including high-performing transfer learning techniques, more sophisticated language models, and newer approaches to content understanding. There are hundreds more papers in NLP, NLU, and NLG which we have not covered in this summary, but we hope for this article to give you a solid foundational understanding of the key papers of 2018.

Artificial Intelligence, thanks to a myriad of dystopian literature, has always been viewed with negative connotations, even though it's still hundreds of years away before total superiority, according to some people's calculations. Despite this, in the last few years, AI, and deep machine learning have gained truly significant, commercial application. As seen in the below chart from MIT, the adoption of AI by business has been happening at a dizzying pace. A pace that is expected to continue. The reasons for this are simple and relatively cyclical in explanation.

Recent advances in deep neural networks (DNNs) owe their success to training algorithms that use backpropagation and gradientdescent. Backpropagation, while highly effective on von Neumann architectures, becomes inefficient when scaling to large networks. Commonly referred to as the weight transport problem, each neuron's dependence on the weights and errors located deeper in the network require exhaustive data movement which presents a key problem in enhancing the performance and energy-efficiency of machine-learning hardware. In this work, we propose a bio-plausible alternative to backpropagation drawing from advances in feedback alignment algorithms in which the error computation at a single synapse reduces to the product of three scalar values, satisfying the three factor rule. Using a sparse feedback matrix, we show that a neuron needs only a fraction of the information previously used by the feedback alignment algorithms to yield results which are competitive with backpropagation. Consequently, memory and compute can be partitioned and distributed whichever way produces the most efficient forward pass so long as a single error can be delivered to each neuron. We evaluate our algorithm using standard data sets, including ImageNet, to address the concern of scaling to challenging problems. Our results show orders of magnitude improvement in data movement and 2 improvement in multiply-and-accumulate operations over backpropagation. All the code and results are available under https://github.com/bcrafton/ssdfa. I. INTRODUCTION The demise of Dennard scaling [11] and decline of Moores Law [27] have exposed the fundamental scaling limitations of the von Neumann models of computing. This transition is accompanied by the realization that in a fast evolving, socially interconnected world, we are observing a seismic shift in the amount of unstructured data that need to be processed in real-time [25] which has heralded the third wave of Artificial Intelligence and the exponential growth of Machine Learning in data-analytics, real-time control, computer vision, robotics and so on. We expect that intelligent systems of the future will be limited by the energy growth of data movement rather than compute.

Recent research on dense captioning based on the recurrent neural network and the convolutional neural network has made a great progress. However, mapping from an image feature space to a description space is a nonlinear and multimodel task, which makes it difficult for the current methods to get accurate results. In this paper, we put forward a novel approach for dense captioning based on hourglass-structured residual learning. Discriminant feature maps are obtained by incorporating dense connected networks and residual learning in our model. Finally, the performance of the approach on the Visual Genome V1.0 dataset and the region labelled MS-COCO (Microsoft Common Objects in Context) dataset are demonstrated. The experimental results have shown that our approach outperforms most current methods.

The symptoms include proximal and axial muscle weakness, distal hyperlaxity, joint contractures, and critical respiratory insufficiency,which requires assisted ventilation and results in a reduced live expectancy. Moreover, the skin and other connective tissues where collagen VI is abundant are also affected [1, 2]. The collagen VI structural defects are related to mutations of three main genes (COL6A1, COL6A2, and COL6A3, OMIM 254090 and 158810). Thus, the new advances in genome editing tools open the possibility to successfully treat these neuromuscular diseases for the first time. This opportunity, though, comes with important challenges. Beyond the challenges of gene editing, this paper focuses on the challenges arising when trying to formally evaluate the efficiency of the therapeutic approaches in the recovery of the collagen VI microfibrillar network.

We propose a general framework for denoising high-dimensional measurements which requires no prior on the signal, no estimate of the noise, and no clean training data. The only assumption is that the noise exhibits statistical independence across different dimensions of the measurement. Moreover, our framework is not restricted to a particular denoising model. We show how it can be used to calibrate any parameterised denoising algorithm, from the single hyperparameter of a median filter to the millions of weights of a deep neural network. We demonstrate this on natural image and microscopy data, where we exploit noise independence between pixels, and on single-cell gene expression data, where we exploit independence between detections of individual molecules. Finally, we prove a theoretical lower bound on the performance of an optimal denoiser. This framework generalizes recent work on training neural nets from noisy images and on cross-validation for matrix factorization.