Benchmarking Amazon EMR vs Databricks

At Insider, we use Apache Spark as the primary data processing engine to mine our clients' clickstream data and feed ML-ready data into our machine learning pipelines to enable personalizations. We have been using Spark since version 1.5 and always looking for ways to improve efficiency. If you are interested too, check out our blog post about how Spark 3 reduced our Amazon EMR cost by 40%. To further improve our platform's efficiency, we decided to conduct a trial with the Databricks platform. Before moving forward with the Databricks platform and the benchmarks, let's see how we utilize Apache Spark and Amazon EMR, and the pain points to understand better our current solutions and challenges.