Our VB-LoRA achieves higher scores with significantly smaller number ofstoredparameters.

Amidst therecent strides inevaluating LargeLanguage Models forCode (Code LLMs), existing benchmarks havemainly focused onthefunctional correctness of generated code, neglecting the importance of their computational efficiency.

Large language models (LLMs) have shown impressive performance on downstream tasks by in-contextlearning (ICL), which heavily relies on the quality of demonstrations selected from a large set of annotated examples.