ABSTRACT We explore active learning (AL) utterance selection for improving the accuracy of new underrepresented domains in a natural language understanding (NLU) system. Moreover, we propose an AL algorithm called Majority-CRF that uses an ensemble of classification and sequence labeling models to guide utterance selection for annotation. Experiments with three domains show that Majority-CRF achieves 6.6%-9% relative error rate reduction compared to random sampling with the same annotation budget, and statistically significant improvements compared to other AL approaches. Additionally, case studies with human-in-the-loop AL on six new domains show 4.6%-9% improvement on an existing NLU system. Index Terms-- Active Learning, Domain Expansion, Natural Language Understanding, Intelligent Virtual Assistants 1. INTRODUCTION Intelligent virtual assistants (IVA) with natural language understanding (NLU), such as Amazon Alexa, Apple Siri, Google Assistant, and Microsoft Cortana, are becoming increasingly popular. For IVA, NLU is a distinct component of spoken language understanding (SLU) [1], in conjunction with automatic speech recognition (ASR) and dialog management (DM). ASR produces a token sequence from speech, which is passed to NLU for both classifying the action or "intent" that the user wants to invoke (e.g.

The reconfigurability, energy-efficiency, and massive parallelism on FPGAs make them one of the best choices for implementing efficient deep learning accelerators. However, state-of-art implementations seldom consider the balance between high throughput of computation power and the ability of the memory subsystem to support it. In this paper, we implement an accelerator on FPGA by combining the sparse Winograd convolution, clusters of small-scale systolic arrays, and a tailored memory layout design. We also provide an analytical model analysis for the general Winograd convolution algorithm as a design reference. Experimental results on VGG16 show that it achieves very high computational resource utilization, 20x ~ 30x energy efficiency, and more than 5x speedup compared with the dense implementation.

We construct a general unified framework for learning representation of structured data, i.e. data which cannot be represented as the fixed-length vectors (e.g. sets, graphs, texts or images of varying sizes). The key factor is played by an intermediate network called SAN (Set Aggregating Network), which maps a structured object to a fixed length vector in a high dimensional latent space. Our main theoretical result shows that for sufficiently large dimension of the latent space, SAN is capable of learning a unique representation for every input example. Experiments demonstrate that replacing pooling operation by SAN in convolutional networks leads to better results in classifying images with different sizes. Moreover, its direct application to text and graph data allows to obtain results close to SOTA, by simpler networks with smaller number of parameters than competitive models.

T. Poggio observes, analyzes and predicts the evolution of deep learning from both mathematical and biological sides(which is the focus in our article) in [1]"Deep learning: mathematics and neuroscience". He mentions that, "it is telling that several of the algorithmic tricks that were touted as breakthroughs just a couple of years ago are now regarded as unnecessary ", while " some of the other ideas " such as residual learning " are more fundamental" "and likely to be more durable, though their exact form is bound to change somewhat " . In a word, he predicts that residual learning is a more durable component within the evolution of deep learning.

The application of deep learning to symbolic domains remains an active research endeavour. Graph neural networks (GNN), consisting of trained neural modules which can be arranged in different topologies at run time, are sound alternatives to tackle relational problems which lend themselves to graph representations. In this paper, we show that GNNs are capable of multitask learning, which can be naturally enforced by training the model to refine a single set of multidimensional embeddings $\in \mathbb{R}^d$ and decode them into multiple outputs by connecting MLPs at the end of the pipeline. We demonstrate the multitask learning capability of the model in the relevant relational problem of estimating network centrality measures, i.e. is vertex $v_1$ more central than vertex $v_2$ given centrality $c$?. We then show that a GNN can be trained to develop a $lingua$ $franca$ of vertex embeddings from which all relevant information about any of the trained centrality measures can be decoded. The proposed model achieves $89\%$ accuracy on a test dataset of random instances with up to 128 vertices and is shown to generalise to larger problem sizes. The model is also shown to obtain reasonable accuracy on a dataset of real world instances with up to 4k vertices, vastly surpassing the sizes of the largest instances with which the model was trained ($n=128$). Finally, we believe that our contributions attest to the potential of GNNs in symbolic domains in general and in relational learning in particular.

In this work, we attempt to answer a critical question: whether there exists some input sequence that will cause a well-trained discrete-space neural network sequence-to-sequence (seq2seq) model to generate egregious outputs (aggressive, malicious, attacking, etc.). And if such inputs exist, how to find them efficiently. We adopt an empirical methodology, in which we first create lists of egregious output sequences, and then design a discrete optimization algorithm to find input sequences that will cause the model to generate them. Moreover, the optimization algorithm is enhanced for large vocabulary search and constrained to search for input sequences that are likely to be input by real-world users. In our experiments, we apply this approach to dialogue response generation models trained on three real-world dialogue data-sets: Ubuntu, Switchboard and OpenSubtitles, testing whether the model can generate malicious responses. We demonstrate that given the trigger inputs our algorithm finds, a significant number of malicious sentences are assigned large probability by the model, which reveals an undesirable consequence of standard seq2seq training. Recently, research on adversarial attacks (Goodfellow et al., 2014; Szegedy et al., 2013) has been gaining increasing attention: it has been found that for trained deep neural networks (DNNs), when an imperceptible perturbation is applied to the input, the output of the model can change significantly (from correct to incorrect). This line of research has serious implications for our understanding of deep learning models and how we can apply them securely in real-world applications. It has also motivated researchers to design new models or training procedures (Madry et al., 2017), to make the model more robust to those attacks.

Google and Facebook are teaming up to make each company's artificial intelligence technologies work better together. The two companies said Tuesday that an unspecified number of engineers are collaborating to make Facebook's open source machine learning PyTorch framework work with Google's custom computer chips for machine learning, dubbed Tensor Processing Units, or TPU. "Today, we're pleased to announce that engineers on Google's TPU team are actively collaborating with core PyTorch developers to connect PyTorch to Cloud TPUs," Google Cloud director of product management Rajen Sheth wrote in a blog post. "The long-term goal is to enable everyone to enjoy the simplicity and flexibility of PyTorch while benefiting from the performance, scalability, and cost-efficiency of Cloud TPUs." Facebook product manager for artificial intelligence Joseph Spisak said in a separate blog post that "Engineers on Google's Cloud TPU team are in active collaboration with our PyTorch team to enable support for PyTorch 1.0 models on this custom hardware."

The fundamental idea behind a neural network is to simulate multiple interconnected cells within a computer's "brain" so that it can learn from its environment, recognize different patterns and, in general, make decisions similar to a human being. A basic neural network contains about millions of artificial neurons known as units. These units are arranged in layers with each layer connecting to the other side. Each Hidden Unit is a squeezed linear function of its inputs. Output Units -- These signal how the Network should respond to the information it has recently acquired.

Could The Transformer be another nail in the coffin for RNNs? Doing away with the clunky for loops, it finds a way to allow whole sentences to simultaneously enter the network in batches. The miracle is that NLP can then reclaim the advantage of python's highly efficient linear algebra libraries. The time-saving is then spent deploying more layers into the model. So far it seems the result is faster convergence and better results.

If the math behind data science is an enigma, the Khan Academy is a great place for insight. There are courses for different levels, and Sal Khan's relaxed delivery will get you through even the most difficult concepts (I think I have a small crush on him after the hours I've spent listening to his narratives!).