Differentiating through the Fr\'echet Mean