Deep neural networks (DNNs) often require good regularizers to generalize well. Currently, state-of-the-art DNN regularization techniques consist in randomly dropping units and/or connections on each iteration of the training algorithm. Dropout and DropConnect are characteristic examples of such regularizers, that are widely popular among practitioners. However, a drawback of such approaches consists in the fact that their postulated probability of random unit/connection omission is a constant that must be heuristically selected based on the obtained performance in some validation set. To alleviate this burden, in this paper we regard the DNN regularization problem from a Bayesian inference perspective: We impose a sparsity-inducing prior over the network synaptic weights, where the sparsity is induced by a set of Bernoulli-distributed binary variables with Beta (hyper-)priors over their prior parameters. This way, we eventually allow for marginalizing over the DNN synaptic connectivity for output generation, thus giving rise to an effective, heuristics-free, network regularization scheme. We perform Bayesian inference for the resulting hierarchical model by means of an efficient Black-Box Variational inference scheme. We exhibit the advantages of our method over existing approaches by conducting an extensive experimental evaluation using benchmark datasets.

To train an inference network jointly with a deep generative topic model, making it both scalable to big corpora and fast in out-of-sample prediction, we develop Weibull hybrid autoencoding inference (WHAI) for deep latent Dirichlet allocation, which infers posterior samples via a hybrid of stochastic-gradient MCMC and autoencoding variational Bayes. The generative network of WHAI has a hierarchy of gamma distributions, while the inference network of WHAI is a Weibull upward-downward variational autoencoder, which integrates a deterministic-upward deep neural network, and a stochastic-downward deep generative model based on a hierarchy of Weibull distributions. The Weibull distribution can be used to well approximate a gamma distribution with an analytic Kullback-Leibler divergence, and has a simple reparameterization via the uniform noise, which help efficiently compute the gradients of the evidence lower bound with respect to the parameters of the inference network. The effectiveness and efficiency of WHAI are illustrated with experiments on big corpora.

Recent deep learning based denoisers are trained to minimize the mean squared error (MSE) between the output of a network and the ground truth noiseless image in the training data. Thus, it is crucial to have high quality noiseless training data for high performance denoisers. Unfortunately, in some application areas such as medical imaging, it is expensive or even infeasible to acquire such a clean ground truth image. We propose a Stein's Unbiased Risk Estimator (SURE) based method for training deep learning based denoisers without ground truth data. We demonstrated that our SURE based method only with noisy input data was able to train CNN based denoising networks that yielded performance close to that of the original MSE based deep learning denoisers with ground truth data.

In recent years, analyzing task-based fMRI (tfMRI) data has become an essential tool for understanding brain function and networks. However, due to the sheer size of tfMRI data, its intrinsic complex structure, and lack of ground truth of underlying neural activities, modeling tfMRI data is hard and challenging. Previously proposed data-modeling methods including Independent Component Analysis (ICA) and Sparse Dictionary Learning only provided a weakly established model based on blind source separation under the strong assumption that original fMRI signals could be linearly decomposed into time series components with corresponding spatial maps. Meanwhile, analyzing and learning a large amount of tfMRI data from a variety of subjects has been shown to be very demanding but yet challenging even with technological advances in computational hardware. Given the Convolutional Neural Network (CNN), a robust method for learning high-level abstractions from low-level data such as tfMRI time series, in this work we propose a fast and scalable novel framework for distributed deep Convolutional Autoencoder model. This model aims to both learn the complex hierarchical structure of the tfMRI data and to leverage the processing power of multiple GPUs in a distributed fashion. To implement such a model, we have created an enhanced processing pipeline on the top of Apache Spark and Tensorflow library, leveraging from a very large cluster of GPU machines. Experimental data from applying the model on the Human Connectome Project (HCP) show that the proposed model is efficient and scalable toward tfMRI big data analytics, thus enabling data-driven extraction of hierarchical neuroscientific information from massive fMRI big data in the future.

We propose SEARNN, a novel training algorithm for recurrent neural networks (RNNs) inspired by the "learning to search" (L2S) approach to structured prediction. RNNs have been widely successful in structured prediction applications such as machine translation or parsing, and are commonly trained using maximum likelihood estimation (MLE). Unfortunately, this training loss is not always an appropriate surrogate for the test error: by only maximizing the ground truth probability, it fails to exploit the wealth of information offered by structured losses. Further, it introduces discrepancies between training and predicting (such as exposure bias) that may hurt test performance. Instead, SEARNN leverages test-alike search space exploration to introduce global-local losses that are closer to the test error. We first demonstrate improved performance over MLE on two different tasks: OCR and spelling correction. Then, we propose a subsampling strategy to enable SEARNN to scale to large vocabulary sizes. This allows us to validate the benefits of our approach on a machine translation task.

Predictive Analytics World for Industry 4.0 is coming to Munich, 12-13 Jun 2018. Find the latest trends and technologies in machine & deep learning for the era of Internet of Things and artificial intelligence. Super Early Bird Rates end Mar 2.

"Deep learning" has become a hot topic in the general rush to launch AI products. But many of these products will fail because companies are putting branding ahead of functionality. Success depends on understanding what deep learning is, how it works, and what its most effective applications are. Traditional machine learning algorithms are typically linear, in that they can be represented by only one node that linearly transforms input to output. Previously called artificial neural networks or neural networks, deep learning uses multiple such nodes, organized like the neural networks originally invented in 1943 to model how human brains work.

SenseTime Co., a Chinese artificial intelligence start-up backed by Alibaba Holdings Ltd., has formed an alliance with one of the US' top university to jointly explore human and machine intelligence. The Beijing-based firm will jointly work with the Massachusetts Institute of Technology undertake research into original AI technologies such as computer-vision, human-intelligence-inspired algorithms, medical imaging, and robotics, online news outlet QQ Tech reported. SenseTime recently became the first company to join MIT's Intelligence Quest, which aims to leverage the Institute's strengths in brain and cognitive science and computer science to advance research into human and machine intelligence. Considered the world's leading AI unicorn valued at more than USD3 billion, SenseTime has developed a sophisticated proprietary deep learning platform and built applications for multiple industries. The company has offices in Hong Kong, Beijing, Shenzhen, Hangzhou, Shanghai, Chengdu, Kyoto, Tokyo and Singapore.

In this article, I cover the 12 types of AI problems i.e. I address the question: in which scenarios should you use Artificial Intelligence (AI)? Recently, I conducted a strategy workshop for a group of senior executives running a large multi national. In the workshop, one person asked the question: How many cats does it need to identify a Cat? This question is in reference to Andrew Ng's famous paper on Deep Learning where he was correctly able to identify images of Cats from YouTube videos.

In this post, we implement the famous word embedding model: word2vec. Here are the paper and the original code by C. Word2vec is so classical ans widely used. However, it's implemented with pure C code and the gradient are computed manually. Nowadays, we get deep-learning libraries like Tensorflow and PyTorch, so here we show how to implement it with PyTorch. Actually, original word2vec implemented two models, skip-gram and CBOW.