Decoding with Limited Teacher Supervision Requires Understanding When to Trust the Teacher