Goto

Collaborating Authors

An AI backed by Elon Musk just 'evolved' to learn by itself

#artificialintelligence

Most of today's artificial intelligence (AI) systems rely on machine learning algorithms that can predict specific outcomes by drawing on pre-established values, but now researchers from OpenAI, a company funded by no less than Elon Musk and Peter Thiel, who are trying to democratise AI for "human good" just discovered – literally – that a machine learning system they created to predict the next character in the text of reviews from Amazon evolved into an unsupervised learning system that could learn how to read sentiment. That's a pretty big deal, and it's also something that, at the moment, even the researchers themselves can't explain. "We were very surprised that our model learned an interpretable feature, and that simply predicting the next character in Amazon reviews resulted in discovering the concept of sentiment," said OpenAI in a blog. According to the post OpenAI's neural network was able to train itself and analyse sentiment accurately by classifying Amazon's reviews as either positive or negative – and it then generated follow on text that fit with the sentiment. The AI the team used was what's known as a multiplicative long short-term memory (LSTM) model that was trained for a month, processing 12,500 characters a second using Nvidia Pascal GPU's – which Nvidia's own CEO gifted to Elon Musk last year – with "4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chunk of text."


An AI backed by Elon Musk just 'evolved' to learn by itself

#artificialintelligence

Most of today's artificial intelligence (AI) systems rely on machine learning algorithms that can predict specific outcomes by drawing on pre-established values, but now researchers from OpenAI, a company funded by no less than Elon Musk and Peter Thiel, who are trying to democratise AI for "human good" just discovered – literally – that a machine learning system they created to predict the next character in the text of reviews from Amazon evolved into an unsupervised learning system that could learn how to read sentiment. That's a pretty big deal, and it's also something that, at the moment, even the researchers themselves can't explain. "We were very surprised that our model learned an interpretable feature, and that simply predicting the next character in Amazon reviews resulted in discovering the concept of sentiment," said OpenAI in a blog. According to the post OpenAI's neural network was able to train itself and analyse sentiment accurately by classifying Amazon's reviews as either positive or negative – and it then generated follow on text that fit with the sentiment. The AI the team used was what's known as a multiplicative long short-term memory (LSTM) model that was trained for a month, processing 12,500 characters a second using Nvidia Pascal GPU's – which Nvidia's own CEO gifted to Elon Musk last year – with "4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chunk of text."


Unsupervised Transfer Learning via BERT Neuron Selection

arXiv.org Machine Learning

Recent advancements in language representation models such as BERT have led to a rapid improvement in numerous natural language processing tasks. However, language models usually consist of a few hundred million trainable parameters with embedding space distributed across multiple layers, thus making them challenging to be fine-tuned for a specific task or to be transferred to a new domain. To determine whether there are task-specific neurons that can be exploited for unsupervised transfer learning, we introduce a method for selecting the most important neurons to solve a specific classification task. This algorithm is further extended to multi-source transfer learning by computing the importance of neurons for several single-source transfer learning scenarios between different subsets of data sources. Besides, a task-specific fingerprint for each data source is obtained based on the percentage of the selected neurons in each layer. We perform extensive experiments in unsupervised transfer learning for sentiment analysis, natural language inference and sentence similarity, and compare our results with the existing literature and baselines. Significantly, we found that the source and target data sources with higher degrees of similarity between their task-specific fingerprints demonstrate a better transferability property. We conclude that our method can lead to better performance using just a few hundred task-specific and interpretable neurons.


A Hierarchical Aspect-Sentiment Model for Online Reviews

AAAI Conferences

To help users quickly understand the major opinions from massive online reviews, it is important to automatically reveal the latent structure of the aspects, sentiment polarities, and the association between them. However, there is little work available to do this effectively. In this paper, we propose a hierarchical aspect sentiment model (HASM) to discover a hierarchical structure of aspect-based sentiments from unlabeled online reviews. In HASM, the whole structure is a tree. Each node itself is a two-level tree, whose root represents an aspect and the children represent the sentiment polarities associated with it. Each aspect or sentiment polarity is modeled as a distribution of words. To automatically extract both the structure and parameters of the tree, we use a Bayesian nonparametric model, recursive Chinese Restaurant Process (rCRP), as the prior and jointly infer the aspect-sentiment tree from the review texts. Experiments on two real datasets show that our model is comparable to two other hierarchical topic models in terms of quantitative measures of topic trees. It is also shown that our model achieves better sentence-level classification accuracy than previously proposed aspect-sentiment joint models.


Learning Sentiment-Specific Word Embedding via Global Sentiment Representation

AAAI Conferences

Context-based word embedding learning approaches can model rich semantic and syntactic information. However, it is problematic for sentiment analysis because the words with similar contexts but opposite sentiment polarities, such as good and bad, are mapped into close word vectors in the embedding space. Recently, some sentiment embedding learning methods have been proposed, but most of them are designed to work well on sentence-level texts. Directly applying those models to document-level texts often leads to unsatisfied results. To address this issue, we present a sentiment-specific word embedding learning architecture that utilizes local context informationas well as global sentiment representation. The architecture is applicable for both sentence-level and document-level texts. We take global sentiment representation as a simple average of word embeddings in the text, and use a corruption strategy as a sentiment-dependent regularization. Extensive experiments conducted on several benchmark datasets demonstrate that the proposed architecture outperforms the state-of-the-art methods for sentiment classification.