Improving Classification Performance With Human Feedback: Label a few, we label the rest