Finetuning vision foundation models often improves in-domain accuracy but comes at the cost of robustness under distribution shift. We revisit Mixout, a stochastic regularizer that intermittently replaces finetuned weights with their pre-trained reference, through the lens of a single-run, weight-sharing implicit ensemble. This perspective reveals three key levers that govern robustness: the masking anchor, resampling frequency, and mask sparsity. Guided by this analysis, we introduce GMixout, which (i) replaces the fixed anchor with an exponential moving-average snapshot that adapts during training, and (ii) regulates masking period via an explicit resampling-frequency hyperparameter. Experiments on benchmarks covering covariate shift, corruption, and class imbalance, ImageNet / ImageNet-L T, DomainNet, iWildCam, and CIFAR100-C, GMixout consistently improves in-domain accuracy beyond zero-shot performance while surpassing both Model Soups and strong parameter-efficient finetuning baselines under distribution shift. A core assumption in standard machine/deep learning is that training and test examples are independently and identically distributed (i.i.d.) (V apnik, 1991). However, in practice, real-world datasets often violate this assumption due to shifts in input marginal distributions (Koh et al., 2021; Gul-rajani & Lopez-Paz, 2021), spurious correlations between inputs and labels (Geirhos et al., 2020), and imbalanced class distributions (Y ang et al., 2023; Liu et al., 2019). The advent of foundation models opens new opportunities for addressing distribution shifts (Bommasani, 2021; Radford et al., 2021; Oquab et al., 2023). Notably, they often perform well even in zero-shot settings (Radford et al., 2021), underscoring how large-scale, heterogeneous pretraining can alleviate many challenges.