[D] Transfer Learning Models with Unintuitive results

I don't know why but I've had a hard time figuring this out, or even figuring out what I should do to figure it out. So, I've got 3 different data sets, A, B and C. I want to build some cross domain models with them (they are all a different domain with the same feature set). The method for evaluation is iterated k-fold cross validation. So, I'll split my data into 5 pieces and do cross validation, get a result, and then get a different split. At the end I average all of the results for a given model together.