FairLRF: Achieving Fairness through Sparse Low Rank Factorization