Learning Chess Blindfolded: Evaluating Language Models on State Tracking

Recently, transformer-based language models have stretched notions of what is possible with the simple self-supervised objective of language modeling, becoming a fixture in state of the art language technologies [Vaswani et al., 2017, Devlin et al., 2019, Brown et al., 2020]. However, the black box nature of these models combined with the complexity of natural language makes it challenging to measure how accurately they represent the world state underlying the text. In order to better measure the extent to which these models can capture the world state underlying the symbolic data they consume, we propose training and studying transformer language models for the game of chess. Chess provides a simple, constrained, and deterministic domain where the exact world state is known. Chess games can also be transcribed exactly and unambiguously using chess notations (Section 2). Most importantly, the form of chess notations allows us to probe our language models for aspects of the board state using simple prompts (Section 3) and without changing the language modeling objective or introducing any new classifiers.