Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

Abstract--Humour is a substantial element of human social behaviour, affect, and cognition. Its automatic understanding can facilitate a more naturalistic human-device interaction and the humanisation of artificial intelligence. Current methods of humour detection have been exclusively based on staged data making them inadequate for'real-world' applications. We contribute to addressing this deficiency by introducing the novel Passau-Spontaneous Football Coach Humour (Passau-SFCH) dataset, comprising about 11 hours of recordings. The Passau-SFCH dataset is annotated for the presence of humour and its dimensions (sentiment and direction) as proposed in Martin's Humor Style Questionnaire. We conduct a series of experiments, employing pretrained Transformers, convolutional neural networks, and expert-designed features. The performance of each modality (text, audio, video) for spontaneous humour recognition is analysed and their complementarity is investigated. Our findings suggest that for the automatic analysis of humour and its sentiment, facial expressions are most promising, while humour direction can be best modelled via text-based features. The results reveal considerable differences among different subjects, highlighting the individuality and contextuality of humour usage and style. Further, we observe that a decision-level fusion yields the best recognition result. The Passau-SFCH dataset is available upon request.