Analyzing Social Biases in Japanese Large Language Models

BBQ (Parrish et al., 2022) is a Question Answering (QA) dataset to assess With the development of Large Language Models whether models can correctly understand the context (LLMs) across languages, there is a growing interest of various social categories, and is widely in the extent to which models exhibit social used to evaluate social biases in LLMs. We describe biases against diverse categories. Various social the details of BBQ in Section 3. CrowS-bias benchmarks have been provided (Rudinger Pairs (Nangia et al., 2020) is a dataset for analyzing et al., 2018; Zhao et al., 2018; Nangia et al., 2020; the social biases of masked language models Li et al., 2020; Nadeem et al., 2021; Dhamala et al., with fill-in-the-blank questions about social categories.