Chip vendors implement new applications in CPUs. If the application is suitable for GPUs and DSPs, it may move to them next. Over time, companies develop ASICs and ASSPs. Is Deep learning is moving through the same sequence? In the brief history of deep neural networks (DNNs), users have tried several hardware archi tec tures to increase their performance.

Much of our curriculum is based on feedback from corporate and government partners about the technologies they are using and learning. In addition to their feedback we wanted to develop a data-driven approach for determining what we should be teaching in our data science corporate training and our free fellowship for masters and PhDs looking to enter data science careers in industry. Below is a ranking of 23 open-source deep learning libraries that are useful for Data Science, based on Github and Stack Overflow activity, as well as Google search results. The table shows standardized scores, where a value of 1 means one standard deviation above average (average score of 0). For example, Caffe is one standard deviation above average in Github activity, while deeplearning4j is close to average.

Romeo Kienzler takes the reader on a big and detailed tour through significant Spark topics and exercises, which occur in the practical usage of Spark in Big Data, Analytics, Data Science and Analytic Data Warehouse ("ADW") projects. In his book topics like the new Spark V2 Ecosystem, Machine Learning, Spark Streaming, Graph Processing, Cluster Design and Management (Yarn and Mesos), Cloud based deployments, Performance topics around HDFS, Date importing and handling, Spark Data Source API, Spark Dataframes and Datasets API, Code Generation for expression evaluation, Project Tungsten, Spark error handling and much more are covered. If you have taken one or more of the well done Spark courses from Databricks before, the topics might familiar but the book covers even some more enhanced topics as well it can be taken as a good comprehension or as in-depth notes. Additionally the book focus on very specific details and problems in parallel programming with Spark, derived from practical use cases.As well the book contains links and references on papers, literature and web forums. To summarize I would recommend this book as an excellent starting point and Spark reference guide.

In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress ($>50$\%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

In an era when big data are becoming the norm, there is less concern with the quantity but more with the quality and completeness of the data. In many disciplines, data are collected from heterogeneous sources, resulting in multi-view or multi-modal datasets. The missing data problem has been challenging to address in multi-view data analysis. Especially, when certain samples miss an entire view of data, it creates the missing view problem. Classic multiple imputations or matrix completion methods are hardly effective here when no information can be based on in the specific view to impute data for such samples. The commonly-used simple method of removing samples with a missing view can dramatically reduce sample size, thus diminishing the statistical power of a subsequent analysis. In this paper, we propose a novel approach for view imputation via generative adversarial networks (GANs), which we name by VIGAN. This approach first treats each view as a separate domain and identifies domain-to-domain mappings via a GAN using randomly-sampled data from each view, and then employs a multi-modal denoising autoencoder (DAE) to reconstruct the missing view from the GAN outputs based on paired data across the views. Then, by optimizing the GAN and DAE jointly, our model enables the knowledge integration for domain mappings and view correspondences to effectively recover the missing view. Empirical results on benchmark datasets validate the VIGAN approach by comparing against the state of the art. The evaluation of VIGAN in a genetic study of substance use disorders further proves the effectiveness and usability of this approach in life science.

In this paper, we use variational recurrent neural network to investigate the anomaly detection problem on graph time series. The temporal correlation is modeled by the combination of recurrent neural network (RNN) and variational inference (VI), while the spatial information is captured by the graph convolutional network. In order to incorporate external factors, we use feature extractor to augment the transition of latent variables, which can learn the influence of external factors. With the target function as accumulative ELBO, it is easy to extend this model to on-line method. The experimental study on traffic flow data shows the detection capability of the proposed method.

Lifelong learning is the problem of learning multiple consecutive tasks in a sequential manner where knowledge gained from previous tasks is retained and used for future learning. It is essential towards the development of intelligent machines that can adapt to their surroundings. In this work we focus on a lifelong learning approach to generative modeling where we continuously incorporate newly observed streaming distributions into our learnt model. We do so through a student-teacher architecture which allows us to learn and preserve all the distributions seen so far without the need to retain the past data nor the past models. Through the introduction of a novel cross-model regularizer, the student model leverages the information learnt by the teacher, which acts as a summary of everything seen till now. The regularizer has the additional benefit of reducing the effect of catastrophic interference that appears when we learn over streaming data. We demonstrate its efficacy on streaming distributions as well as its ability to learn a common latent representation across a complex transfer learning scenario.

There has recently been significant interest in hard attention models for tasks such as object recognition, visual captioning and speech recognition. Hard attention can offer benefits over soft attention such as decreased computational cost, but training hard attention models can be difficult because of the discrete latent variables they introduce. Previous work used REINFORCE and Q-learning to approach these issues, but those methods can provide high-variance gradient estimates and be slow to train. In this paper, we tackle the problem of learning hard attention for a sequential task using variational inference methods, specifically the recently introduced VIMCO and NVIL. Furthermore, we propose a novel baseline that adapts VIMCO to this setting. We demonstrate our method on a phoneme recognition task in clean and noisy environments and show that our method outperforms REINFORCE, with the difference being greater for a more complicated task.

The future of technology is artificial intelligence, but before it can radically change the world, we need to develop proper hardware that can give the machines the ability to learn. To that end, The Wall Street Journal reports that Intel is working with Facebook and a number of other companies on a new processor called Nervana --also known as the Nervana Neural Network Processor. The chip is the result of work done by Intel Nervana, and was designed to accelerate the AI learning process known as deep learning, in which a computer is taught to recognize objects, patterns, speech, and complex concepts so that it can mimic the same learning capabilities exhibited by humans. Intel's partners are helping "fine tune" Nervana, while Facebook has started taking a deeper look at the processor and realizing its capabilities. "[They said], 'Hey, this really could change the way we think about artificial intelligence. And help us really steer how we build software and hardware," said Intel Chief Executive Brian Krzanich at The WSJ D.Live conference on Tuesday.

Intellectsoft has been offering AI-based software solutions, so we have started a series of blog posts to shed a light on what AI is, its applications, as well as how to implement it successfully in the enterprise. The previous post was a case-driven guide to machine learning. Deep learning is set to take us to a technologically advanced, automated future of self-driving cars and robotic assistants. However, what it is and how it works still remains a subject significantly more complex than most users imagine. Join us, as we take a closer look at deep learning without going to the neighboring territories of mathematics and software engineering.