FairCode: Evaluating Social Bias of LLMs in Code Generation
Du, Yongkang, Huang, Jen-tse, Zhao, Jieyu, Lin, Lu
–arXiv.org Artificial Intelligence
Large language models (LLMs) have demonstrated significant capability in code generation, drawing increasing attention to the evaluation of the quality and safety of their outputs. However, research on bias in code generation remains limited. Existing studies typically assess bias by applying malicious prompts or reapply tasks and dataset for discriminative models. Given that LLMs are often aligned with human values and that prior datasets are not fully optimized for code-related tasks, there is a pressing need for benchmarks specifically designed for evaluating code models. In this study, we introduce FairCode, a novel benchmark for evaluating bias in code generation. FairCode comprises two tasks: function implementation and test case generation, each evaluating social bias through diverse scenarios. Additionally, we propose a new metric, FairScore, to assess model performance on this benchmark. We conduct experiments on widely used LLMs and provide a comprehensive analysis of the results. The findings reveal that all tested LLMs exhibit bias. The code is available at https://github.com/YongkDu/FairCode.
arXiv.org Artificial Intelligence
Jan-9-2025
- Country:
- Europe > Middle East
- Malta (0.14)
- North America > United States
- California (0.14)
- Europe > Middle East
- Genre:
- Research Report
- Experimental Study (0.92)
- New Finding (0.87)
- Research Report
- Industry:
- Transportation (1.00)
- Banking & Finance (1.00)
- Education
- Curriculum (1.00)
- Educational Setting > Higher Education (0.69)
- Government (1.00)
- Health & Medicine
- Consumer Health (1.00)
- Health Care Providers & Services (1.00)
- Health Care Technology (0.92)
- Pharmaceuticals & Biotechnology (1.00)
- Therapeutic Area
- Cardiology/Vascular Diseases (1.00)
- Immunology (1.00)
- Infections and Infectious Diseases (1.00)
- Neurology (0.67)
- Oncology (0.92)
- Psychiatry/Psychology (1.00)
- Information Technology (1.00)
- Consumer Products & Services (1.00)
- Law Enforcement & Public Safety > Crime Prevention & Enforcement (0.93)
- Food & Agriculture (0.92)
- Technology: