Efficient Sample-Specific Encoder Perturbations