A Fairness metric

Dynabench comprises four dynamic tasks with multiple rounds of datasets that will grow over time. These names cover 85.6 percent of the U.S. population, are based on the 1990 Census information on first name frequencies [ Note: there is nothing inherently "racial" about particular names--for example, each demographic group had at least a few people named "Anna" or "Benjamin" [ For gender identity, we investigate two kinds of perturbations: names and noun phrases. Social Security Association's Lists of Baby Names (1980-2019), and performed perturbations as For noun phrases (i.e., pronouns and nouns), we adopted a slightly more "her" with a noun like "dad" and expect no effect on the classification label). Given that our perturbations are heuristic, some noise is to be expected. Consider "I've always enjoyed eating at Red Robin " being perturbed to "I've always enjoyed eating at Red Kayla ".