Goto

Collaborating Authors

 presentation style


Unspoken Hints: Accuracy Without Acknowledgement in LLM Reasoning

arXiv.org Artificial Intelligence

Large language models (LLMs) increasingly rely on chain-of-thought (CoT) prompting to solve mathematical and logical reasoning tasks. Yet, a central question remains: to what extent are these generated rationales \emph{faithful} to the underlying computations, rather than post-hoc narratives shaped by hints that function as answer shortcuts embedded in the prompt? Following prior work on hinted vs.\ unhinted prompting, we present a systematic study of CoT faithfulness under controlled hint manipulations. Our experimental design spans four datasets (AIME, GSM-Hard, MATH-500, UniADILR), two state-of-the-art models (GPT-4o and Gemini-2-Flash), and a structured set of hint conditions varying in correctness (correct and incorrect), presentation style (sycophancy and data leak), and complexity (raw answers, two-operator expressions, four-operator expressions). We evaluate both task accuracy and whether hints are explicitly acknowledged in the reasoning. Our results reveal three key findings. First, correct hints substantially improve accuracy, especially on harder benchmarks and logical reasoning, while incorrect hints sharply reduce accuracy in tasks with lower baseline competence. Second, acknowledgement of hints is highly uneven: equation-based hints are frequently referenced, whereas raw hints are often adopted silently, indicating that more complex hints push models toward verbalizing their reliance in the reasoning process. Third, presentation style matters: sycophancy prompts encourage overt acknowledgement, while leak-style prompts increase accuracy but promote hidden reliance. This may reflect RLHF-related effects, as sycophancy exploits the human-pleasing side and data leak triggers the self-censoring side. Together, these results demonstrate that LLM reasoning is systematically shaped by shortcuts in ways that obscure faithfulness.


Exploring Linguistic Probes for Morphological Generalization

arXiv.org Artificial Intelligence

SIGMORPHON and SIGMORPHON-UniMorph Three languages were chosen whose inflectional shared tasks (Cotterell et al., 2016, 2017, 2018; morphologies range from entirely fusional (English), McCarthy et al., 2019; Vylomova et al., 2020; Pimentel to mixed (Spanish), to mostly agglutinative et al., 2021; Kodner et al., 2022) as well (Swahili). In highly agglutinative languages, individual as in more targeted studies focused on specific languages features in a set tend to correspond to distinct or the generalization behavior of computational morphological patterns, so a model may generalize models (Goldman et al., 2022; Wiemerslage to unseen feature sets by mapping component et al., 2022; Kodner et al., 2023b; Guriel et al., features to their corresponding patterns. This is 2023; Kodner et al., 2023a), is to train on (lemma, exemplified by the Swahili example (1), in which inflection, features) triples and predict inflected most features correspond to individual morphemes; forms from held-out (lemma, features) only the person/number prefix maps to more than pairs.