According to the Big Five theory of personality, personality traits can be organized into five primary dimensions, including extraversion, agreeableness, conscientiousness, neuroticism, and openness to experience. These dimensions are associated with myriad life outcomes, such as job satisfaction or health. Soto conducted a replication study of 78 previously published personality–life outcome findings after high-profile failures by others to replicate studies in other areas of psychology. Of personality–life outcome effects, 87% replicated successfully with effect sizes that were 77% as large as those in the original studies. Replicability was predicted by features of the original studies and the replication studies.
Leonard Martin agrees with Strack's concerns and says the replicators didn't fully follow their procedure. The work was so sloppy, he argued via email, that "the real story here may not be about the replicability of the pen in the mouth study or the replicability of psychology research in general but about the current method of assessing replicability." Given that such efforts can alter established findings in the field and tarnish people's reputations, he said that "Project Replication" should be very careful: "If the current lack of rigor continues, then psychology may find itself in its own version of the McCarthy era." Strack had one more concern: "What I really find very deplorable is that this entire replication thing doesn't have a research question." It does "not have a specific hypothesis, so it's very difficult to draw any conclusions," he told me.
Last year the United States Food and Drug Administration (FDA) cleared a total of 12 AI tools that use machine learning for health (ML4H) algorithms to inform medical diagnosis and treatment for patients. The tools are now allowed to be marketed, with millions of potential users in the US alone.Because ML4H tools directly affect human health, their development from experiments in labs to deployment in hospitals progresses under heavy scrutiny. A critical component of this process is reproducibility. A team of researchers from MIT, University of Toronto, New York University, and Evidation Health have proposed a number of "recommendations to data providers, academic publishers, and the ML4H research community in order to promote reproducible research moving forward" in their new paper Reproducibility in Machine Learning for Health. Just as boxers show their strength in the ring by getting up again after being knocked to the canvas, researchers test their strength in the arena of science by ensuring their work's reproducibility.
The replicability of some scientific findings has recently been called into question. To contribute data about replicability in economics, we replicated 18 studies published in the American Economic Review and the Quarterly Journal of Economics between 2011 and 2014. All of these replications followed predefined analysis plans that were made publicly available beforehand, and they all have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We found a significant effect in the same direction as in the original study for 11 replications (61%); on average, the replicated effect size is 66% of the original. The replicability rate varies between 67% and 78% for four additional replicability indicators, including a prediction market measure of peer beliefs.
Reproducibility and replicability are cornerstones of scientific inquiry. Although there is some debate on terminology and definitions, if something is reproducible, it means that the same result can be recreated by following a specific set of steps with a consistent dataset. If something is replicable, it means that the same conclusions or outcomes can be found using slightly different data or processes. Without reproducibility, process and findings can't be verified. Without replicability, it is difficult to trust the findings of a single study.