The Limited Impact of Medical Adaptation of Large Language and Vision-Language Models