According to a recent open letter, society needs to immediately pause development of "giant" AI models, or risk apocalyptic outcomes. Massive job losses, the destruction of consensus reality and even the end of all organic life on Earth have all been mooted as risks of pressing forward with development of these systems before we understand their intricacies. The high-water mark of these is GPT-4, the snappily named AI that underpins the latest version of the breakthrough ChatGPT service. Creating anything more powerful than GPT-4, before we spend at least six months working out its limits and risks, would be too dangerous, more than 1,000 AI experts say. I decided to spend some time with the new ChatGPT myself.

This work was done in collaboration with Ding Ding and Sergey Ermolin from Intel. In recent years, the scale of datasets and models used in deep learning has increased dramatically. Although larger datasets and models can improve the accuracy in many AI applications, they often take much longer to train on a single machine. However, it is not very common to distribute the training to large clusters using current popular deep learning frameworks, compared to what's been long around in the Big Data area, as it's often harder to gain access to a large GPU cluster and lack of convenient facilities in popular DL frameworks for distributed training. By leveraging the cluster distribution capabilities in Apache Spark, BigDL successfully performs very large-scale distributed training and inference.