Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order gradients only). Gating mechanisms that use softly saturating activation functions to emulate the discrete switching of digital logic circuits are good examples of this. We propose to exploit the injection of appropriate noise so that the gradients may flow easily, even if the noiseless application of the activation function would yield zero gradient. Large noise will dominate the noise-free gradient and allow stochastic gradient descent to explore more. By adding noise only to the problematic parts of the activation function, we allow the optimization procedure to explore the boundary between the degenerate (saturating) and the well-behaved parts of the activation function. We also establish connections to simulated annealing, when the amount of noise is annealed down, making it easier to optimize hard objective functions. We find experimentally that replacing such saturating activation functions by noisy variants helps training in many contexts, yielding state-of-the-art or competitive results on different datasets and task, especially when training seems to be the most difficult, e.g., when curriculum learning is necessary to obtain good results.

Nonnegative matrix factorization (NMF) is a popular dimension reduction technique that produces interpretable decomposition of the data into parts. However, this decompostion is not generally identifiable (even up to permutation and scaling). While other studies have provide criteria under which NMF is identifiable, we present the first (to our knowledge) characterization of the non-identifiability of NMF. We describe exactly when and how non-uniqueness can occur, which has important implications for algorithms to efficiently discover alternate solutions, if they exist.

The complex graph structure of the Web - with different relations or edge types - has motivated a large body of research tackling the challenge of mining multi-relational data in the presence of noise, partial inconsistencies, ambiguities, or duplicate entities. State-of-the-art advances in this field are relevant to many applications such as link prediction [1], Resource Description Framework (RDF) mining [2], entity linking [3], recommender systems [4], and natural language processing [5]. However, new paradigms are still needed for statistical and computational inference for very large multi-relational datasets, like the ones produced at massive scale in projects such as the Google's Knowledge Graph [6], YAGO [7], and in Semantic Web initiatives such as DBpedia [8]. Factorization models are considered state-of-the-art approaches for Statistical Relational Learning (SRL) in which they have exhibited a high predictive performance [9], [10], [11]. Factorization models for multi-relational data associate entities and relations with latent feature vectors and model predictions about unknown relationships through operations on these vectors (e.g., dot products). Optimizing the predictions for a number of relations can be seen as a prediction task with multiple target variables. For example, multi-target models can support information retrieval tasks in Linked Open Data bases like DBPedia by providing estimates of facts, that are neither explicitly stated in the knowledge base nor can be inferred from logical entailment, enabling probabilistic queries on such databases [1], [2]. Another example in the context of social web recommender systems, is that such services are not only interested in recommending, for instance, news items to a user but also recommending other users as potential new friends. State-of-the-art factorization models approach the multitarget prediction task by sharing the parameters used for all target relations.

We present an algorithm, called the Offset Tree, for learning to make decisions in situations where the payoff of only one choice is observed, rather than all choices. The algorithm reduces this setting to binary classification, allowing one to reuse of any existing, fully supervised binary classification algorithm in this partial information setting. We show that the Offset Tree is an optimal reduction to binary classification. In particular, it has regret at most $(k-1)$ times the regret of the binary classifier it uses (where $k$ is the number of choices), and no reduction to binary classification can do better. This reduction is also computationally optimal, both at training and test time, requiring just $O(\log_2 k)$ work to train on an example or make a prediction. Experiments with the Offset Tree show that it generally performs better than several alternative approaches.

"Nothing can have value without being an object of utility." Karl Marx Marx's words have unexpected application in the world of commercial technology – essentially the takeaway is that the success of any technology trend or movement can only be proven by how "useful" it proves over time. Let's examine one of the technology trends dominating the airwaves today, Machine Learning, through the lens of "utility". BCC research has predicted that the market for Machine Learning will grow at a compounded annual rate of close to 20% and cross 15 Billion by 2019. Presumably with numbers like that, there's no denying the prevalence of the technology trend – just what kind of utility is it really delivering, though?

Next time you ask Google for directions or run an image search, thank Le Viet Quoc. The 34-year-old Vietnamese engineer is part of the team behind Google Brain, an artificial intelligence (AI) research project whose technology is responsible for such features, reports VnExpress. Part of Google's not-so-secret research outfit X, which pioneers cutting-edge technology like self-driving cars and delivery drones, Quoc works in a field known as "deep learning" which uses the human brain as a model to create "neural networks" for computers. Though deep learning's development has been slow, engineers like Quoc are making progress: in 2012, Google Brain made headlines when its network of 16,000 computer processors successfully learned how to search for cat videos on YouTube, despite being given no information prior to the test on how to identify such animals. The Stanford grad, who holds a doctorate in computer science and was named one of MIT's Innovators Under 35, is still working toward the creation of better, more intelligent machines.

Welcome to the second lesson in our Practicing Deep Learning Series. Thoughtly is writing a multi-part tutorial series focused on understanding the foundations of Deep Learning, specifically as they apply to Natural Language Processing. If you want to jump to another post check the post listing here. Last time we focused on the elements of a simple single neuron network. We specifically discussed those that feed into the neuron – the inputs and weights – and their interaction via the dot product.

I went to school in Cambridge University in England, then went to Stanford to do my master's around 2000. I met up with Sergey Brin around then when Google was a tiny company and he invited me to join Google. So I started working part-time for Google. It was a fantastic time at the company--200 people, one building, and bright, idealistic, change-the-world kind of people. Naturally, when I finished my master's I joined full-time.

The Second National Data Science Bowl, a data science competition where the goal was to automatically determine cardiac volumes from MRI scans, has just ended. We participated with a team of 4 members from the Data Science lab at Ghent University in Belgium and finished 2nd! The team kunsthart (artificial heart in English) consisted of Ira Korshunova, Jeroen Burms, Jonas Degrave (@317070), 3 PhD students, and professor Joni Dambre. It's also a follow-up of last year's team Deep Sea, which finished in first place for the First National Data Science Bowl. This blog post is going to be long, here is a clickable overview of different sections. The goal of this year's Data Science Bowl was to estimate minimum (end-systolic) and maximum (end-diastolic) volumes of the left ventricle from a set of MRI-images taken over one heartbeat. These volumes are used by practitioners to compute an ejection fraction: fraction of outbound blood pumped from the heart with each heartbeat.

April 1, 2016 – Boston, MA – Cloud-based computing and artificial intelligence represent the future of content creation, distribution, public relations, and marketing. SHIFT Communications, the premiere data-driven PR agency, announced today the release of the Heuristic And Recurrent Ontological Lexicon Deep-learner, or HAROLD, the world's first artificially intelligent (AI), cloud-based PR employee. HAROLD's creation represents the first AI employee of a virtual public relations workforce. HAROLD is based on the proven TensorFlow multidimensional data array artificial intelligence software, first developed by Google Brain, part of Google's Machine Intelligence division. SHIFT Vice President of Marketing Technology Christopher Penn said, "HAROLD provides SHIFT with limitless scale. Your standard PR team has 10-15 humans; with a cloud-based AI employee, we can create a million new'employees' in seconds. It would take your average account team several weeks to pitch a 5,000 person media list. HAROLD can replicate itself and call the whole media list simultaneously."