Computational and Statistical Hardness of Calibration Distance