Multi-Scale Representation Learning for Protein Fitness Prediction