Distilling What We Know