Perception and Semantic Aware Regularization for Sequential Confidence Calibration