Measurement in the Age of LLMs: An Application to Ideological Scaling

Social science pertains to complex constructs denoted by terms like "ideology", "power", or "culture", whose meanings are contextual and generally hard to pin down precisely. Although slippery and subjective, such terms are routinely used in conversation, among experts and non-experts alike, without anyone (except the occasional pedant) demanding formal definitions from their conversational partners. It is indeed a feature of natural language discourse that such terms are assumed to wear many hats, and that conversational partners must cooperate to arrive at mutually intelligible meanings. This cooperation is typically tacit, and speakers coordinate on a shared meaning by offering examples, reformulations, and engaging generally in an elaborative process that builds upon shared context and common knowledge. In so doing however, speakers inevitably introduce new terms requiring their own processes of disambiguation.