Which of these 2 techniques is most appropriate to create a hold-out set?
You almost certainly should do (a) subject-wise cross-validation rather than (b) record-wise cross-validation. In some sense, an independent observation is a different subject. If you want to predict performance on new subjects, you must test on subjects you didn't train on! In typical settings, repeated observations of the same individual are correlated with each other even after conditioning on features. Hence with record-wise cross-validation, your test set isn't independent of your training set!
Oct-13-2016, 20:51:13 GMT
- Technology: