The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning