Recently, as shown by Papernot and McDaniel (2018), feeding in neighbors of the target as input can improve the robustness of data representations and help the model generalize better to out-of-distribution samples.
Fine-tuning is a common practice in deep learning, achieving excellent generalization results on downstream tasks using relatively little training data.