Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis