As part of my Technology and Innovation MBA program at Ted Rogers School of Management, I took a data and knowledge management course which teaches students the principles and practices of knowledge management. The second part of the course delves on tools used in data management and analytics. Although the theoretical part of the course was a bit dry, the hands-on portion was very interesting and exposed students to several different tools to capture, clean and analyze data. One of the tasks given to students was to capture and analyze twitter data. Although students had access to Netlytics, which is a neat cloud-based text and social network analysis tool that also collects Twitter data, students were encouraged to find other ways to collect Twitter data.

One of the most difficult challenges reporting and analytics face in public relations measurement is sentiment analysis. Machines attempt textual analysis of sentiment all the time; more often than not, it goes horribly wrong. How does it go wrong? Machines are incapable of understanding context. Machines are typically programmed to look for certain keywords as proxies for sentiment.

So you report with reasonable accuracies what the sentiment about a particular brand or product is. After publishing this report, your client comes back to you and says "Hey this is good. Now can you tell me ways in which I can convert the negative sentiments into positive sentiments?" – Sentiment Analysis stops there and we enter the realms of Opinion Mining. Opinion Mining is about having a deeper understanding of the review that was written. Typically, a detailed review will not just have a sentiment attached to it. It will have information and valuable feedback that can literally help to build the next strategy.

As a part of Twitter Data Analysis, So far I have completed Movie review using R& Document Classification using R. Today we will be dealing with discovering topics in Tweets, i.e. to mine the tweets data to discover underlying topics– approach known as Topic Modeling. A statistical approach for discovering "abstracts/topics" from a collection of text documents based on statistics of each word. In simple terms, the process of looking into a large collection of documents, identifying clusters of words and grouping them together based on similarity and identifying patterns in the clusters appearing in multitude. When we apply Topic Modeling to the above statements, we will be able to group statement 1&2 as Topic-1 (later we can identify that the topic is Sport),statement 3 as Topic-2 (topic is Movies), statement 4&5 as Topic-3 (topic isdata Analytics). Topic Modeling can be achieved by using Latent Dirichlet Allocation algorithm.

This is the last – for now – installment of my mini-series on sentiment analysis of the Stanford collection of IMDB reviews (originally published on recurrentnull.wordpress.com). So far, we've had a look at classical bag-of-words models and word vectors (word2vec). We saw that from the classifiers used, logistic regression performed best, be it in combination with bag-of-words or word2vec. We also saw that while the word2vec model did in fact model semantic dimensions, it was less successful for classification than bag-of-words, and we explained that by the averaging of word vectors we had to perform to obtain input features on review (not word) level. So the question now is: How would distributed representations perform if we did not have to throw away information by averaging word vectors?

This long article with a lot of source code was posted by Suraj V Vidyadaran. Suraj is pursuing a Master in Computer Science at Temple university primarily focused in Data Science specialization. His areas of interests are in sentiment analysis, data visualization, big data and machine learning. I was surprised to see the overlap with our recent article on top 10 machine learning algorithms. You can read the full article (with voluminous source code in R) here.

This is the continuation of my mini-series on sentiment analysis of movie reviews, which originally appeared on recurrentnull.wordpress.com. Last time, we had a look at how well classical bag-of-words models worked for classification of the Stanford collection of IMDB reviews. As it turned out, the "winner" was Logistic Regression, using both unigrams and bigrams for classification. The best classification accuracy obtained was .89 So, bag-of-words models may be surprisingly successful, but they are limited in what they can do.

Imagine I show you a book review, on amazon.com, Imagine I hide the number of stars, – all you get to see is the number of stars. And now I'm asking you, that review, is it good or bad? Well, it should be easy, for humans (although depending on the input there can be lots of disagreement between humans, too.) But if you want to do it automatically, it turns out to be surprisingly difficult.

But you probably know this already, if only from the preponderance of conference presentations, blogs and trade articles on the topic. Yes, text analytics are all the rage these days. You may feel under the gun to catch up, but if you're late to the game, you may be comforted to know that for many people, text analytics aren't living up to the hype. Nearly every researcher I come in contact with at conferences and through my professional network is at least actively investigating text analysis if they haven't already adopted a solution. And in either case, they're frequently underwhelmed. It's my experience that there are two primary reasons for this: How Do I Know Before I Buy?

Natural Language Processing (NLP) is a vast area of Computer Science that is concerned with the interaction between Computers and Human Language[1]. Within NLP many tasks are – or can be reformulated as – classification tasks. In classification tasks we are trying to produce a classification function which can give the correlation between a certain'feature' and a class . This Classifier first has to be trained with a training dataset, and then it can be used to actually classify documents. Training means that we have to determine its model parameters.