Using OpenNLP for Named-Entity-Recognition in Scala - DZone Big Data

A common challenge in Natural Language Processing (NLP) is Named Entity Recognition (NER) - this is the process of extracting specific pieces of data from a body of text, commonly people, places and organizations (for example trying to extract the name of all people mentioned in a wikipedia article). NER is a problem that has been tackled many times over the evolution of NLP, from dictionary-based, to rule-based, to statistical models and more recently using Neural Nets to solve the problem. Whilst there have been recent attempts to crack the problem without it, the crux of the issue is really that for approach to learn it needs a large corpus of marked up training data (there are some marked up corpora available, but the problem is still quite domain specific, so training on the WSJ data might not perform particularly well against your domain specific data) and finding a set of 100,000 marked up sentences is no easy feat. There are some approaches that can be used to tackle this by generating training data - but it can be hard to generate truly representative data and so this approach always risks over-fitting to the generated data. Having previously looked at Stanford's NLP library for some sentiment analysis, this time I am looking at using the OpenNLP library Further to this, the Stanford library is licensed under GPL which makes it harder to use in any kind of commercial/startup setting.