Human Judgment on Humor Expressions in a Community-Based Question-Answering Service

For understanding humorous dialogue, a collection of humorous expressions is needed. In addition to humorous expressions, their annotations are important to be used as language resources. In this paper, we analyzed how human assessors annotate humorous expressions extracted from an online community-based question-answering (CQA) corpus, which contains many interesting examples of humorous communication. We analyzed the annotation results of a collection of humorous expressions as done by 28 annotators in terms of the degree of humor and categorization of humor. We found the assessments to be quite subjective, and only marginal inter-annotator agreements were observed. This result suggests that the variability in humor annotations is not noise resulting from erroneous assessment but is rooted in personality differences of the annotators. It would be necessary to incorporate the individual differences in humor perception for properly utilizing the resources. We discuss the possibility to improve the collection process by applying filtering techniques.