Big data experts talk about text, Twitter and turning quantamental

Using machines to read text as a way to enhance understanding of market movements is a topic of intense polarisation and debate. Back in the 90s, work on natural language processing (NLP) involved teams of linguists and computer scientists attempting to code up rules of grammar. Recent work has focused on techniques like word embedding, the underlying idea that a word is characterised by the company it keeps; semantic similarities between words are based on their distribution in large samples of data. The "bag of words" approach has been applied commercially in finance for more than 10 years. But it can depend on the source of information being analysed: a rule-based approach can work pretty well for news articles that follow certain editorial processes, while social media proves much more challenging.