Attributing Culture-Conditioned Generations to Pretraining Corpora

Recent works show that these biases may stem from uneven cultural representation in pretraining corpora. This work investigates how pretraining leads to biased culture-conditioned generations by analyzing how models associate entities with cultures based on pretraining data patterns. Additionally, the model favors generating entities with extraordinarily high frequency regardless of the conditioned culture, reflecting biases toward frequent pretraining terms irrespective of relevance. Our findings reflect trends observed specifically within OLMo-7B's pretraining data and are limited to this dataset. We make no claims about whether these results reflect real-world conditions.] In open-ended generative tasks like narrative writing or dialogue, language models often show bias against marginalized social groups based on gender, race, or culture (Gallegos et al., 2024; Manvi et al., 2024; Li et al., 2024b). Cultural bias is particularly notable due to the vast number of cultures to account for. Cultures are often unevenly represented in the pretraining corpora, with some mentioned more frequently than others, irrespective of their real-world prevalence (Li et al., 2024a). Recent studies reveal that models favor entities (Naous et al., 2023) and opinions (Ryan et al., 2024) from frequently represented cultures in pretraining while showing inadequate knowledge and templated answers for less frequent ones (Li et al., 2024b). Such biases in culture-conditioned generations can be linked to studies showing that LLMs' memorization and generalization are constrained by pretraining data imbalances. Zhang et al. (2024) find that these imbalances cause models to overgeneralize to high-frequency knowledge, overshadowing lower-frequency knowledge.