Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges

The application of machine learning (ML) in detecting, diagnosing, and treating mental health disorders is garnering increasi ng attention. Traditionally, research has focused on single modalities, such as text from clinical notes, audio from speech samples, or video of interaction patterns. Recently, multimodal ML, which combines information from multiple modalities, has demonstrated significant promise in offering novel insights into human behavior patterns and recognizing mental health symptoms and risk factors. Despite its potential, multimodal ML in mental health remains an emerging field, facing several complex challenges before practical applications can be effectively developed. This survey provides a comprehensive overview of the data availability a nd current state-of-the-art multimodal ML applications for mental health. It discusses key challenges that must be addressed to advance the field.