Large scale matrix multiplication with pyspark (or -- how to match two large datasets of company…

Spark and pyspark have wonderful support for reliable distribution and parallelization of programs as well as support for many basic algebraic operations and machine learning algorithms. In this post we describe the motivation and means of performing name-by-name matching of two large datasets of company names using Spark. Our goal is to match two large sets of company names. We're looking at two long lists of company names, list A and list B and we aim to match companies from A to companies from B. In this example our goal is to match both GOOGLE INC. and Google, inc (from list A) to Google (from list B); and to match MEDIUM.COM to Medium Inc; and Amazon labs to Amazon, etc… OK, At first we thought we'd try the most simple and trivial solution, to see how well it works, if not for anything else, at the very least in order to establish a baseline for future attempts. The most simple thing to do is just case-insensitive string equation test.