Perplexity Cannot Always Tell Right from Wrong