Bayesian Low-Rank Factorization for Robust Model Adaptation