Introducing the Natural Language Processing Library for Apache Spark - The Databricks Blog

Our second core requirement was frictionless reuse of existing Spark libraries. Part of it is our own pet peeve – why does every NLP library out there have to build its own topic modeling and word embedding implementations? The other part is pragmatic – we're a small team under tight deadlines and need to make the most of what's already there. When we started thinking about a Spark NLP library, we first asked Databricks to point us to whoever is already building one. When the answer came there there isn't one, the next ask was to help us make sure the design and API of the library fully meet Spark ML's API guidelines.