Dealing with imbalanced data: undersampling, oversampling and proper cross-validation

Inside the cross-validation loop, get a sample out and do not use it for anything related to features selection, oversampling or model building. Oversample your minority class, without the sample you already excluded. Use the excluded sample for validation, and the oversampled minority class the majority class, to create the model. Repeat n times, where n is your number of samples (if doing leave one participant out cross-validation). Inside the cross-validation loop, get a sample out and do not use it for anything related to features selection, oversampling or model building. Oversample your minority class, without the sample you already excluded. Use the excluded sample for validation, and the oversampled minority class the majority class, to create the model. Repeat n times, where n is your number of samples (if doing leave one participant out cross-validation).