Calibrated and Robust Foundation Models for Vision-Language and Medical Image Tasks Under Distribution Shift