On-the-Job Learning with Bayesian Decision Theory