7 Ways to Handle Large Data Files for Machine Learning - Machine Learning Mastery

Exploring and applying machine learning algorithms to datasets that are too large to fit into memory is pretty common. In this post, I want to offer some common suggestions you may want to consider. Some machine learning tools or libraries may be limited by a default memory configuration. Check if you can re-configure your tool or library to allocate more memory. A good example is Weka, where you can increase the memory as a parameter when starting the application. Are you sure you need to work with all of the data?