PerMod: Perceptually Grounded Voice Modification with Latent Diffusion Models