We do a hyperparameter search for LoRA over the following variables: LoRA dropout { 0.0, 0.05, LoRA α is always proportional to the learning rate. We find that LoRA dropout 0.05 is useful for small models (7B, 13B), but not for larger models (33B, We use the same preprocessing of the Super-Natural Instruction dataset as Wang et al. RA finetuning experiments outlined in Section 5. This limits the dataset to 9,209 examples. HH-RLHF This is a human preference dataset about helpfulness and harmlessness.

Deep learning algorithms have shown significant development thanks tothelargepre-collected dataset, such as SQuAD [47]innatural language processing (NLP), and ImageNet [4] in computer vision.

However, if error is heavy-tailed, some policies obtain arbitrarily high reward despite achieving no more utility than the base model-a phenomenon we call catastrophic Goodhart. We adapt a discrete optimization method to measure the tails of reward models, finding that they are consistent with light-tailed error.