Developing a Machine Learning Model to QA human meta data attribution • /r/MachineLearning

I want to develop a machine learning model in R that I can deploy in Java. I want to describe what I've tried, and how it failed, and what my next iteration is so every one here can help guide me. We have a system where a human will examine raw text and assign it meta-data where each piece of meta-data is a separate category. For example, say we had String 1 and available categories A-Z and a human assigned String 1 categories A, B, C, and F. There is a large amount of human QA that happens afterwards to ensure that String 1 either received all the categories it should have and didn't receive any categories it shouldn't have, for example String 1 should not have received the f category. I am tasked with developing a way to automatically detect if a String needs QA after meta-data has been assigned to it.