Approaching the Harm of Gradient Attacks While Only Flipping Labels