LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arena

LLM safety and ethical alignment are widely discussed, but the impact of content moderation on user satisfaction remains underexplored. To address this, we analyze nearly 50,000 Chatbot Arena response - pairs using a novel fine - tuned RoBER T a model, that we trained on hand - labeled data to disentangle refusals due to ethical concerns from other refusals due to technical disabilities or lack of information. Our findings reveal a significant refusal penalty on content moderation, with users choosing ethical - based refusals roughly one - fourth as often as their preferred LLM response compared to standard responses . However, the context and phrasing play critical roles: refusals on highly sensitive prompts, such as illegal content, achieve higher win rates than less sensitive ethical concerns, and longer responses closely aligned with the prompt perform better. These results emphasize the need for nuanced moderation strategies that balance ethical safeguards with user satisfaction. Moreover, we find that the refusal penalty is notably lower in evaluations using the LLM - as - a - Judge method, highlighting discrepancies be tween user and automated assessments. Trigger Warning and Disclaimer: This paper discusses content moderation in LLMs, including sensitive topics such as hate speech, harassment, and illegal activities, as part of an analysis of LLM performance and user satisfaction. The study does not endorse or promote any harmful, illegal, or unethical content, nor does it make any normative judgments about the " right amount " of content moderation.