Active Learning Behind The Scenes

In my previous posts, I described what is Active Learning (introduction to Active Learning and the main approaches) and how one can implement such a pipeline in a simple and generic way (architecture for Active Learning pipeline). This post is based on a talk I gave at Reversim Summit 2021. I want to jump a few steps forward and talk about how I evaluate my Active Learning models' results and how it helped me to catch a huge bug. In general, when we talk about Active Learning, we talk about creating a Data Selector. An automatic algorithm that will be able to choose a subset from our unlabeled data and will give our model the largest performance gain, so it will be able to learn and improve its accuracy.