Analyzing and Improving Speaker Similarity Assessment for Speech Synthesis