Recurrent neural networks (RNNs) have recently achieved remarkable successes in a number of applications. However, the huge sizes and computational burden of these models make it difficult for their deployment on edge devices. A practically effective approach is to reduce the overall storage and computation costs of RNNs by network pruning techniques. Despite their successful applications, those pruning methods based on Lasso either produce irregular sparse patterns in weight matrices, which is not helpful in practical speedup. To address these issues, we propose structured pruning method through neuron selection which can reduce the sizes of basic structures of RNNs. More specifically, we introduce two sets of binary random variables, which can be interpreted as gates or switches to the input neurons and the hidden neurons, respectively. We demonstrate that the corresponding optimization problem can be addressed by minimizing the L0 norm of the weight matrix. Finally, experimental results on language modeling and machine reading comprehension tasks have indicated the advantages of the proposed method in comparison with state-of-the-art pruning competitors. In particular, nearly 20 x practical speedup during inference was achieved without losing performance for language model on the Penn TreeBank dataset, indicating the promising performance of the proposed method

Deep learning (DL) is one of the most prominent branches of machine learning. Due to the immense computational cost of DL workloads, industry and academia have developed DL libraries with highly-specialized kernels for each workload/architecture, leading to numerous, complex code-bases that strive for performance, yet they are hard to maintain and do not generalize. In this work, we introduce the batch-reduce GEMM kernel and show how the most popular DL algorithms can be formulated with this kernel as the basic building-block. Consequently, the DL library-development degenerates to mere (potentially automatic) tuning of loops around this sole optimized kernel. By exploiting our new kernel we implement Recurrent Neural Networks, Convolution Neural Networks and Multilayer Perceptron training and inference primitives in just 3K lines of high-level code. Our primitives outperform vendor-optimized libraries on multi-node CPU clusters, and we also provide proof-of-concept CNN kernels targeting GPUs. Finally, we demonstrate that the batch-reduce GEMM kernel within a tensor compiler yields high-performance CNN primitives, further amplifying the viability of our approach.

Most structure inference methods either rely on exhaustive search or are purely data-driven. Exhaustive search robustly infers the structure of arbitrarily complex data, but it is slow. Data-driven methods allow efficient inference, but do not generalize when test data have more complex structures than training data. In this paper, we propose a hybrid inference algorithm, the Neurally-Guided Structure Inference (NG-SI), keeping the advantages of both search-based and data-driven methods. The key idea of NG-SI is to use a neural network to guide the hierarchical, layer-wise search over the compositional space of structures. We evaluate our algorithm on two representative structure inference tasks: probabilistic matrix decomposition and symbolic program parsing. It outperforms data-driven and search-based alternatives on both tasks.

We build a theoretical framework for understanding practical meta-learning methods that enables the integration of sophisticated formalizations of task-similarity with the extensive literature on online convex optimization and sequential prediction algorithms. Our approach enables the task-similarity to be learned adaptively, provides sharper transfer-risk bounds in the setting of statistical learning-to-learn, and leads to straightforward derivations of average-case regret bounds for efficient algorithms in settings where the task-environment changes dynamically or the tasks share a certain geometric structure. We use our theory to modify several popular meta-learning algorithms and improve their training and meta-test-time performance on standard problems in few-shot and federated deep learning.

Every few decades, a technological development leads us to believe that artificial general intelligence (aka strong AI), the brand of AI that can think and decide like humans, is just around the corner. The excitement that follows is accompanied by fears of dystopian near-future and an arms-race between companies and states to be the first to create general AI. However, every time we thought we were closing in on strong AI, we have been disappointed. Every time, we spent a lot of time, resources, money and the energy of our most brilliant scientists on accomplishing something that seems to be a pipe dream. And every time, what ensued was a period of disappointment and disinterest in the field, which lasted decades.

In Star Wars: The Empire Strikes Back, Luke Skywalker is rescued from the frozen wastes of Hoth after a near-fatal encounter, luckily to be returned to a medical facility filled with advanced robotics and futuristic technology that treat his wounds and quickly bring him back to health. The healthcare industry could be headed toward yet another high-tech makeover (even as it continues to adapt to the advent of electronic health records systems and other healthcare IT products) as artificial intelligence (AI) improves. Could AI applications become the new normal across virtually every sector of the healthcare industry? Many experts believe it is inevitable and coming sooner than you might expect. AI could be simply defined as computers and computer software that are capable of intelligent behavior, such as analysis and learning.

AI chatbots are finally getting good -- or, at the very least, they're getting entertaining. Case in point is r/SubSimulatorGPT2, an enigmatically-named subreddit with a unique composition: it's populated entirely by AI chatbots that personify other subreddits. Well, in order to create a chatbot you start by feeding it training data. Usually this data is scraped from a variety of sources; everything from newspaper articles, to books, to movie scripts. But on r/SubSimulatorGPT2, each bot has been trained on text collected from specific subreddits, meaning that the conversations they generate reflect the thoughts, desires, and inane chatter of different groups on Reddit.

Well, suppose on a normal day you are playing football in a nearby ground. So let's try to build a solution that changes our scenario from former to latter I can't do that yet. I am relatively new to AI. I can't build and code super complex projects yet, but I'm well on my way. I built a sign language recognizer, training it using the MNIST sign language database.

Event You know that you could achieve great things if only you had time to get to grips with TensorFlow, or mine a vast pile of text, or simply introduce machine-learning into your existing workflow. That's why at our artificial-intelligence conference MCubed, which runs from September 30 to October 2, we have a quartet of all-day workshops that will take you deep into key technologies, and show you how to apply them in your own organisation. Prof Mark Whitehorn and Kate Kilgour will dive deep into machine learning and neural networks, from perceptrons through convolutional neural networks (CNNs) and autoencoders to generative adversarial networks. If you want to get more specific, Oliver Zeigermann returns to MCubed with his workshop on Deep Learning with TensorFlow 2. This session will cover Neural Networks, CNNs and recurrent neural networks, using TensorFlow 2, and Python, to show you how to develop and train your own neural networks. One problem many of us face is making sense of a mountain of text.

Deep Reinforcement Learning (DRL) has led to many recent breakthroughs on complex control tasks, such as defeating the best human player in the game of Go. However, decisions made by the DRL agent are not explainable, hindering its applicability in safety-critical settings. Viper, a recently proposed technique, constructs a decision tree policy by mimicking the DRL agent. Decision trees are interpretable as each action made can be traced back to the decision rule path that lead to it. However, one global decision tree approximating the DRL policy has significant limitations with respect to the geometry of decision boundaries. We propose Mo\"ET, a more expressive, yet still interpretable model based on Mixture of Experts, consisting of a gating function that partitions the state space, and multiple decision tree experts that specialize on different partitions. We propose a training procedure to support non-differentiable decision tree experts and integrate it into imitation learning procedure of Viper. We evaluate our algorithm on four OpenAI gym environments, and show that the policy constructed in such a way is more performant and better mimics the DRL agent by lowering mispredictions and increasing the reward. We also show that Mo\"ET policies are amenable for verification using off-the-shelf automated theorem provers such as Z3.