Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator