Dynamic Syntax Mapping: A New Approach to Unsupervised Syntax Parsing

The advancement of large pretrained language models (LLMs) such as BERT [1] has been a catalyst in enhancing performance across a spectrum of natural language processing (NLP) tasks. This progress has spurred investigations into the mechanics of natural language comprehension within these models, particularly concerning their linguistic capabilities. Building on this foundation, our study tests the hypothesis that LLMs inherently contain syntactic dependencies that can be extracted without the need for additional parameters or external guidance. Existing literature in the realm of syntax has delved into two key areas: (1) the examination of syntactically dependent behaviors in language models, such as their ability to handle complex subject-verb agreement scenarios [2, 3, 4, 5, 6, 7, 8], and (2) the feasibility of retrieving syntactic structures from internal representations or mechanisms within the models [11, 12, 13]. The former approach, while agnostic to specific syntactic theories, lacks the clarity offered by the explicit induction of syntactic structures, which is the focus of our work. In an exemplary study, Hewitt and Manning [11] developed a probe to map word representations into a new vector space, facilitating the application of a maximum spanning tree algorithm (MST) for syntactic parsing.