Record linking with Apache Spark's MLlib & GraphX
Recently a colleague asked me to help her with a data problem, that seemed very straightforward at a glance. She had purchased a small set of data from the chamber of commerce (Kamer van Koophandel: KvK) that contained roughly 50k small sized companies (5–20FTE), which can be hard to find online. She noticed that many of those companies share the same address, which makes sense, because a lot of those companies tend to cluster in business complexes. However she also found that many companies on the same address are in fact 1 company divided over multiple registrations. Which are technically different companies, but for this specific case should be treated as 1 single company with combined work force.
May-25-2017, 18:00:12 GMT
- Technology: