Reproducibility Requires Consolidated Artifacts

A. Missing Artifacts We manually evaluate 142 papers that reproduce previous studies and are published at the open-access peer-reviewed journal ReScience C. We tagged each paper based on the issues the authors faced when reproducing the original work with tags denoting implementation issues, hyperparameter issues, and the responsiveness of the original author. The end result is that only a subset of the original trials are valid, which can lead to biased analysis and unreproducible results. Abstract--Machine learning is facing a'reproducibility crisis' B. Problematic Tooling [1] identifies missing and convoluted artifacts as one of the We quantitatively evaluate the inter-project dependency issue main causes of non-reproducible research. Artifacts include for reproducibility [3]. We mine 132 repositories from the configuration details, details on the methodology, and code.