When Pre-trained Visual Representations Fall Short: Limitations in Visuo-Motor Robot Learning