However,inmanyreal-worldtasks,the learned function is intended to satisfy domain-specific constraints. We focus on monotonicity constraints, which are common and require that the function's output increases with increasing values of specific input features.

However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers.

We take advantage of NF preconditioning and NF based Metropolis-Hastings updates for a faster convergence.

However, if error is heavy-tailed, some policies obtain arbitrarily high reward despite achieving no more utility than the base model-a phenomenon we call catastrophic Goodhart. We adapt a discrete optimization method to measure the tails of reward models, finding that they are consistent with light-tailed error.