Online Control of the False Discovery Rate under "Decision Deadlines"

Scientific discoveries form an ongoing, ever-evolving process. Each new experiment offers an opportunity to suggest new hypotheses based on results that have come before. Traditionally, the hypotheses researchers plan to test in an experiment are prespecified before any data from the experiment is visible, as this facilitates control of either the false discovery rate (FDR; Benjamini and Hochberg, 1995) or the probability of producing any false positives (the familywise error rate, or FWER; see, for example Efron and Hastie, 2016) within that experiment. In contrast to fully prespecified procedures, online procedures test hypotheses sequentially, and allow the results of preliminary tests to inform choices about which hypotheses to focus on in future tests (Foster and Stine, 2008). These procedures typically require that error rates be controlled at every stage of the sequence (e.g., Javanmard and Montanari, 2015; Ramdas et al., 2017).