Seeing is not always believing: The Space of Harmless Perturbations