Making a (Counterfactual) Difference One Rationale at a Time