Fast Model-Selection through Adapting Design of Experiments Maximizing Information Gain

Experimentation in the social sciences is a fundamental tool for understanding the mechanisms and heuristics that underlie human behavior. At the same time, running experiments is a costly process and requires careful design in order to test hypotheses while maximizing statistical power. This experimental design process is often guided by the intuition of scientists conducting the research, and while there are many benefits in relying on the intuition of skilled researchers, there is often a lack of principled, optimal experimental design when choosing which experiment to run (Fisher, 1936; Hill, 1995). Despite best efforts, we often are faced with conflicting results that stall scientific progress, which may be one consequence of under-powered or suboptimal experimental designs. Today, social scientists are witnessing two parallel improvements in experimental design techniques. First, a number of new, online experimentation platforms have begun to address the problem of cost and efficiency of experimentation (e.g., nodeGame (Balietti, 2017), Volunteer Science (Radford et al., 2016)). These online platforms are retooling social science by allowing for rapidly-deployed, large-scale experiments involving participants from around the world.