Troubling Trends in Machine Learning Scholarship

This paper aims to instigate discussion, answering a call for papers from the ICML Machine Learning Debates workshop. While we stand by the points represented here, we do not purport to offer a full or balanced viewpoint or to discuss the overall quality of science in ML. In many aspects, such as reproducibility, the community has advanced standards far beyond what sufficed a decade ago. We note that these arguments are made by us, against us, by insiders offering a critical introspective look, not as sniping outsiders. The ills that we identify are not specific to any individual or institution. We ourselves have fallen into these patterns, and likely will again in the future. Exhibiting one of these patterns doesn't make a paper bad nor does it indict the paper's authors, however we believe that all papers could be made stronger by avoiding these patterns. While we provide concrete examples, our guiding principles are to (i) implicate ourselves, and (ii) to preferentially select from the work of better-established researchers and institutions that we admire, to avoid singling out junior students for whom inclusion in this discussion might have consequences and who lack the opportunity to reply symmetrically. We are grateful to belong to a community that provides sufficient intellectual freedom to allow us to express critical perspectives. In each subsection below, we (i) describe a trend; (ii) provide several examples (as well as positive examples that resist the trend); and (iii) explain the consequences. Pointing to weaknesses in individual papers can be a sensitive topic. To minimize this, we keep examples short and specific.