The External Validity of Combinatorial Samples and Populations

The widely used 'Counterfactual' definition of Causal Effects was derived for unbiasedness and accuracy - and not generalizability. We propose a simple definition for the External Validity (EV) of Interventions, Counterfactual statements and Samples. We use the definition to discuss several issues that have baffled the counterfactual approach to effect estimation: out-of-sample validity, reliance on independence assumptions or estimation, concurrent estimation of many effects and full-models, bias-variance tradeoffs, statistical power, omitted variables, and connections to supervised and explaining techniques. Methodologically, the definition also allow us to replace the parametric and generally ill-posed estimation problems that followed the counterfactual definition by combinatorial enumeration problems on non-experimental samples. We use over 20 contemporary methods and simulations to demonstrate that the approach leads to accuracy gains in standard out-of-sample prediction, intervention effect prediction and causal effect estimation tasks. The COVID19 pandemic highlighted the need for learning solutions to provide general predictions in small samples - many times with missing variables. We also demonstrate applications in this pressing problem.