SpaCy or Spark NLP -- A Benchmarking Comparison

The aim of this article is to run a realistic Natural Language Processing scenario to compare the leading linguistic programming libraries: enterprise-grade John Snow Labs' Spark NLP and Explosion AI's industrial-strength library spaCy, both of which are open-source with commercially permissive licenses. Comparing two different libraries is not as simple as it sounds. Each library has different implementation methods and thus will have different use cases, data pipelines, and characteristics. In this study, a detailed Spark NLP pipeline will be designed and a parallel code mimicking that will be written using spaCy, focusing primarily on runtime speed. We will then compare the results in terms of memory usage, speed, and accuracy.