Are models biased on text without gender-related language?

Have you ever needed to write a recommendation letter for a student or co-worker and struggled to remember their contributions? Or maybe you're applying for a job or internship and don't know how to start your cover letter. Language Models (LMs), like ChatGPT [1], have become useful tools for writing various types of content, including professional documents like recommendation or cover letters [2, 3]. By providing just a few details, such as the person's name, age, gender, and current position, you can quickly generate initial drafts of these documents. With ChatGPT, users can quickly create an initial draft of a reference letter by specifying a few details about the individual.