I ran 80,000 simulations to investigate different p-value adjustments
However, in a surprise to approximately no one who works professionally with data, we do not live in an ideal world. A variety of pressures compel many practitioners to perform tens, hundreds, or even thousands of significance tests on the same data set. Some reasons for doing this are better than others but, independent of even the very best motivations: this practice basically breaks everyday statistics. The assurance of a getting small p-value–that chance alone would spur null differences to appear this distinct only 5%, 1%, 0.1% of the time–is moot when you're playing the odds hundreds, thousands, or tens of thousands of times. A really really big number divided by a big number [or, equivalently here, multiplied by a small proportion] is still a really really big number.
Apr-21-2022, 08:00:39 GMT