From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set