Mitigating Spurious Correlations in Patch-wise Tumor Classification on High-Resolution Multimodal Images