Design-based individual prediction

Valid inference of the unobserved individual prediction errors is a fundamental issue to supervised machine learning, no matter how confident one is about the obtained predictor. An IID model of the prediction errors is commonly assumed for algorithm-based learning, such as random forest, support vector machine or neural network, which could be misleading in situations where the available observations are not obtained in a completely random fashion. We define and develop a design-based approach to individual prediction, which requires the sample for learning to be selected by a probability design. Whether the adopted predictor is selected from an ensemble of models or a weighted average of them, the proposed approach can provide valid inference of the associated risk with respect to the known sampling design, "irrespectively of the unknown properties of the target population studied" (Neyman, 1934).