On Fairness of Low-Rank Adaptation of Large Models