How do GPUs Improve Neural Network Training?

I bet most of us have heard about "GPUs". There have been sayings that GPU is the best investment you can do for gaming. But the technology that once fancies the Gaming Industry, is now a core element of various other realms including Artificial Intelligence, Video Rendering, Healthcare. The electronic giants once used to focus solely on the gaming industry to design their GPUs, but as the corpus of AI is expanding, the demand and specialization of GPUs are increasing exponentially. The increasing attention on Deep Learning has enabled manufacturers to also focus on software rather than just hardware.