Smoke and Mirrors in Causal Downstream Tasks