One way to increase confidence in the outputs of Large Language Models (LLMs) is to support them with reasoning that is clear and easy to check -- a property we call legibility. We study legibility in the context of solving grade-school math problems and show that optimizing chain-of-thought solutions only for answer correctness can make them less legible. To mitigate the loss in legibility, we propose a training algorithm inspired by Prover-Verifier Game from Anil et al. (2021). Our algorithm iteratively trains small verifiers to predict solution correctness, "helpful" provers to produce correct solutions that the verifier accepts, and "sneaky" provers to produce incorrect solutions that fool the verifier. We find that the helpful prover's accuracy and the verifier's robustness to adversarial attacks increase over the course of training. Furthermore, we show that legibility training transfers to time-constrained humans tasked with verifying solution correctness. Over course of LLM training human accuracy increases when checking the helpful prover's solutions, and decreases when checking the sneaky prover's solutions. Hence, training for checkability by small verifiers is a plausible technique for increasing output legibility. Our results suggest legibility training against small verifiers as a practical avenue for increasing legibility of large LLMs to humans, and thus could help with alignment of superhuman models.

Heaven's Vault, a science fiction adventure told with the appealing restraint of an Asimov classic, begins as something of a reluctant manhunt. Your character, Aliya, an orphan who as a young girl was rescued from a planet of slave traders by an esteemed academic, is summoned home to the university where she grew up. There, her adoptive mother beseeches Aliya to find an old friend who has disappeared while undertaking an archaeological treasure hunt. It's an interruption that Aliya, a freelance archaeologist-cum-treasure hunter herself, could do without. Still, through familial loyalty, or more likely a rivalrous interest in whatever treasure the vanished man was hunting, she glumly agrees to the assignment.

The wood is weather-worn, rough edges smoothed by untold hundreds of years spent floating through space, but still a few symbols remain, meticulously carved into the surface. It's a few words, I think, and the wood was part of a prow--the only remaining bit of a long-lost shipwreck. But I can't quite make out what it says. I recognize a few symbols, one for action and one I associate with movement, and the grouping of glyphs that represents "Me." I'm going to have to make some guesses though.