Characterizing the Accuracy - Efficiency Trade-off of Low-rank Decomposition in Language Models