NLP in the Cloud: Measuring the Quality of NLP APIs

Natural Language Processing seems to have become somewhat of a commodity in recent years. More than a few companies have sprung up that offer basic NLP capabilities through a cloud API. If you'd like to know whether a text carries a positive or negative message, or what people or companies it mentions, you can just send it to one of these black boxes, and receive the answer in less than a second. Superficially, all these NLP APIs look more or less the same. Textrazor, AlchemyAPI, Aylien, MeaningCloud and Lexalytics all offer similar services (named entity recognition, sentiment analysis, keyword extraction, topic identification, etc.), and do so through similar interfaces.