Empirical Policy Evaluation with Supergraphs