Mind the Gap: Aligning the Brain with Language Models Requires a Nonlinear and Multimodal Approach