Transformers and ConvNets Using Counterfactual Simulation Testing