A polar coordinate system represents syntax in large language models

Originally formalized with symbolic representations, syntactic trees may also be effectively represented in the activations of large language models (LLMs). Indeed, a "Structural Probe" can find a subspace of neural activations, where syntacticallyrelated words are relatively close to one-another. However, this syntactic code remains incomplete: the distance between the Structural Probe word embeddings can represent the existence but not the type and direction of syntactic relations. Here, we hypothesize that syntactic relations are, in fact, coded by the relative direction between nearby embeddings. To test this hypothesis, we introduce a "Polar Probe" trained to read syntactic relations from both the distance and the direction between word embeddings.