MinorBench: A hand-built benchmark for content-based risks for children

Large Language Models (LLMs) are rapidly entering children's lives -- through parent-driven adoption, schools, and peer networks -- yet current AI ethics and safety research do not adequately address content-related risks specific to minors. In this paper, we highlight these gaps with a real-world case study of an LLMbased chatbot deployed in a middle school setting, revealing how students used and sometimes misused the system. We evaluate six prominent LLMs under different system prompts, demonstrating substantial variability in their childsafety compliance. Our results inform practical steps for more robust, childfocused safety mechanisms and underscore the urgency of tailoring AI systems to safeguard young users. Large Language Models (LLMs) have seen rapid adoption in educational settings, with both teachers and students recognizing their potential for personalized feedback and instant instructional support. Recent surveys indicate that over half of K-12 teachers in some regions now use LLMs for lesson planning, grading assistance, or creative class activities, while approximately onethird of students--some as young as 12--have experimented with such models for schoolwork (Common Sense Media, 2024). However, the emergence of LLMs in schools raises concerns about children's vulnerability. Children are still developing critical thinking skills, often place higher trust in authoritative-sounding answers, and may not fully understand an AI's limitations.