Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation