Training a CNN with the same data but different labels • /r/MachineLearning

I apologize for the ambiguous title, but it was difficult to compile my question into a sentence. I have a large data-set of paintings, and corresponding class labels generated from their medium. I'm not interested in the output class, but rather the 9216 dimension feature vector generated from the Pool5 layer of the network (I'm using AlexNet). Now, when I generate the class labels from the meta-data assosiated with the painting I'm using the least frequent term as it tends to be more telling. As an example, a painting's medium meta-data may be "Oil and Chalk on Paper"; currently, the least frequent term would have been applied as the target label, in this case "Oil".