Decoupled Action Head: Confining Task Knowledge to Conditioning Layers