Minimizing Human Intervention in Online Classification