Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shift