The Few Govern the Many:Unveiling Few-Layer Dominance for Time Series Models