Stop Doing Fragile Research

Here's a story familiar to anyone who does research in data science or machine learning: (1) you have a brand-new idea for a method to analyze data (2) you want to test it, so you start by generating a random dataset or finding a dataset online.(3) You apply your method to the data, but the results are unimpressive. And you introduce a hyperparameter into your method so that you can fine-tune it, until (5) the method eventually starts producing gorgeous results. However, in taking these steps, you have developed a fragile method, one that is sensitive to the choice of dataset and customized hyperparameters. Rather than developing a more general and robust method, you have made the problem easier.