Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an output activation function for breaking the softmax bottleneck without additional parameters. We re-analyze the softmax bottleneck from the perspective of the output set of log-softmax and identify the cause of the softmax bottleneck. On the basis of this analysis, we propose sigsoftmax, which is composed of a multiplication of an exponential function and sigmoid function. Sigsoftmax can break the softmax bottleneck. The experiments on language modeling demonstrate that sigsoftmax and mixture of sigsoftmax outperform softmax and mixture of softmax, respectively.

An algorithmically hard phase was described in a range of inference problems: even if the signal can be reconstructed with a small error from an information theoretic point of view, known algorithms fail unless the noise-to-signal ratio is sufficiently small. This hard phase is typically understood as a metastable branch of the dynamical evolution of message passing algorithms. In this work we study the metastable branch for a prototypical inference problem, the low-rank matrix factorization, that presents a hard phase. We show that for noise-to-signal ratios that are below the information theoretic threshold, the posterior measure is composed of an exponential number of metastable glassy states and we compute their entropy, called the complexity. We show that this glassiness extends even slightly below the algorithmic threshold below which the well-known approximate message passing (AMP) algorithm is able to closely reconstruct the signal. Counter-intuitively, we find that the performance of the AMP algorithm is not improved by taking into account the glassy nature of the hard phase.

We introduce DeepProbLog, a probabilistic logic programming language that incorporates deep learning by means of neural predicates. We show how existing inference and learning techniques can be adapted for the new language. Our experiments demonstrate that DeepProbLog supports (i) both symbolic and subsymbolic representations and inference, (ii) program induction, (iii) probabilistic (logic) programming, and (iv) (deep) learning from examples. To the best of our knowledge, this work is the first to propose a framework where general-purpose neural networks and expressive probabilistic-logical modeling and reasoning are integrated in a way that exploits the full expressiveness and strengths of both worlds and can be trained end-to-end based on examples.

The t-distributed Stochastic Neighbor Embedding (tSNE) algorithm has become in recent years one of the most used and insightful techniques for the exploratory data analysis of high-dimensional data. tSNE reveals clusters of high-dimensional data points at different scales while it requires only minimal tuning of its parameters. Despite these advantages, the computational complexity of the algorithm limits its application to relatively small datasets. To address this problem, several evolutions of tSNE have been developed in recent years, mainly focusing on the scalability of the similarity computations between data points. However, these contributions are insufficient to achieve interactive rates when visualizing the evolution of the tSNE embedding for large datasets. In this work, we present a novel approach to the minimization of the tSNE objective function that heavily relies on modern graphics hardware and has linear computational complexity. Our technique does not only beat the state of the art, but can even be executed on the client side in a browser. We propose to approximate the repulsion forces between data points using adaptive-resolution textures that are drawn at every iteration with WebGL. This approximation allows us to reformulate the tSNE minimization problem as a series of tensor operation that are computed with TensorFlow.js, a JavaScript library for scalable tensor computations.

In a conversation in Rome facilitated by the Vatican last year, Cardinal Gianfranco Ravasi, president of the Pontifical Council for Culture, reportedly weighed whether or not the phrase "artificial intelligence" is an oxymoron and criticized AI scientists' use of the term "electronic person." That came on the heels of a July 2017 roundtable on the topic at the Dominican School of Philosophy and Theology in Berkeley, California, where speakers from business and technology mixed with academics and theologians.

Machine learning and deep learning are increasingly making their way into consumer-related industries. What do these mysterious technologies entail? Should smaller companies invest in machine learning and deep learning? N-iX engineers try to answers these questions. Machine learning combines the principles of computer science and statistics.

For some us, AI is kind of an iffy proposition. To many, it is nebulous enough to seem like it might replace us or our jobs. And the harbingers of this sea change aren't exactly affirming: every other week in the news, self-driving smart cars keep crashing, with injuries and sometimes fatalities. AI generally doesn't seem to be that well-received in mass media, either, like in movies like Minority Report or TV shows like Westworld. Because of all this, the public perception of AI might be on the negative side. A good way to overcome uneasiness, anxiety or fear is simply be learning more about whatever seems to be the issue or problem.

Idaho has been behind the curve in implementing robotic milking, partly because Lely, one of the major vendors, has been migrating its marketing across the country after starting in western Europe and then heading to Canada and the East Coast. "We just started looking at the western region in the last five or six years," Paetz said. "We don't want our infantry to outrun our lines of support." Across the U.S., robots are now being used in 5 percent or fewer of the dairies, he said.

Insurers are increasingly interested in ways to make better use of data in their daily business decisions. Indeed, almost half of the CIOs we have surveyed this year mentioned that they are currently working on analytics including AI, machine learning, etc. It is also interesting to observe that a minority of insurers (5%) is not working on this topic currently. Our recent Model Insurer program helped us better understand what insurers are doing around data, analytics and AI. The awards' winners in this category have leveraged analytics and AI to improve various facets of their business.

If you had an infinite number of monkeys typing at keyboards, could you produce Shakespeare? But how you would know once they'd typed Shakespeare? In this example, monkeys are what are called Generators in AI, and the English student who checks their work to see if they have written Shakespeare (or anything good) is called a Discriminator. These are the two components of an Generative Adversarial Neural Network. Adversarial Neural Networks are oddly named since they actually cooperate to make things.