Scaling Machine Learning - Part 1: Naive approach

Yet another level is distributed computing. It means multiple machines cooperating as single system to reach common goal. Distributed computing is not always the solution, it is actually often misused for problems that could be solved more efficiently on a single machine. It also introduces many new complexities that may not be necessary, including having to worry about concurrency, time, order, message delivery, network latency, consistency, failures or deployment. Companies including Baidu or Google reportedly use optimized single machine implementations or high performance computing utilising GPUs on single machine or supercomputers for machine learning and other expensive algorithms, but TensorFlow or Spark are prime examples of distributed systems used for this puprose.