Machine Learning Magic: How to Speed Up Offline Inference for Large Datasets

In this blog, guest writers Binyang Li (Software Engineer at Microsoft), Qianxi Zhang (Research Software Engineer at Microsoft), describe how to use Alluxio to solve the challenges while running inference at scale. The original content was published on Alluxio's Blog (Disclaimer: The author is a Founding Member @Alluxio). Offline inference, or batch inference, is an approach to run machine learning (ML) inference in a batch mode when processing a large dataset, as opposed to generating predictions in real-time given the input. The offline inference jobs are typically built on top of big data platforms to scale horizontally, and are running on fixed schedules (e.g. Running inference at scale is challenging.