Classification in Spark 2.0: "Input validation failed" and other wondrous tales - Nodalpoint

Spark 2.0 has been released since last July but, despite the numerous improvements and new features, several annoyances still remain and can cause headaches, especially in the Spark machine learning APIs. Today we'll have a look at some of them, inspired by a recent answer of mine in a Stack Overflow question (the question was about Spark 1.6 but, as we'll see, the issue remains in Spark 2.0). We'll first try a simple binary classification problem in PySpark using Spark MLlib, but, before doing so, let's have a look at the current status of the machine learning APIs in Spark 2.0. Spark MLlib was the older machine learning API for Spark, intended to be gradually replaced by the newest Spark ML library; in Spark 2.0 this terminology has changed (enough, in my opinion, to cause unnecessary confusion): now the whole machine learning functionality is termed "MLlib", with the old MLlib being the so-called "RDD-based API", and the (old) Spark ML library now termed the "MLlib DataFrame-based API". The oldest RDD-based API has now entered maintenance mode, heading for gradual deprecation.