multimodal analysis
A Multimodal Framework for Explainable Evaluation of Soft Skills in Educational Environments
Guerrero-Sosa, Jared D. T., Romero, Francisco P., Menéndez-Domínguez, Víctor Hugo, Serrano-Guerrero, Jesus, Montoro-Montarroso, Andres, Olivas, Jose A.
In the rapidly evolving educational landscape, the unbiased assessment of soft skills is a significant challenge, particularly in higher education. This paper presents a fuzzy logic approach that employs a Granular Linguistic Model of Phenomena integrated with multimodal analysis to evaluate soft skills in undergraduate students. By leveraging computational perceptions, this approach enables a structured breakdown of complex soft skill expressions, capturing nuanced behaviours with high granularity and addressing their inherent uncertainties, thereby enhancing interpretability and reliability. Experiments were conducted with undergraduate students using a developed tool that assesses soft skills such as decision-making, communication, and creativity. This tool identifies and quantifies subtle aspects of human interaction, such as facial expressions and gesture recognition. The findings reveal that the framework effectively consolidates multiple data inputs to produce meaningful and consistent assessments of soft skills, showing that integrating multiple modalities into the evaluation process significantly improves the quality of soft skills scores, making the assessment work transparent and understandable to educational stakeholders.
Uncovering the Genetic Basis of Glioblastoma Heterogeneity through Multimodal Analysis of Whole Slide Images and RNA Sequencing Data
Berjaoui, Ahmad, Roussel, Louis, Sanchez, Eduardo Hugo, Moyal, Elizabeth Cohen-Jonathan
Despite being a minor population of cancer cells, the cancer stem cells that are identified in glioblastoma (GSCs) are thought to be the major driving force behind glioblastoma biological heterogeneity and are likely to explain the high rates of glioblastoma recurrence. In the STEMRI clinical trial aiming to study GB heterogeneity and the enrichment of GSC in certain areas defined by multimodal MRI (NCT01872221) [4] different GSC sub-populations extracted from tumor samples obtained by multimodal MRI guided surgery were xenografted into mice brain to study their invasion patterns as well as their aggressiveness. RNA-seq on each tumor bulk samples was also performed. The observed differences in mice survival according to the GSC implanted confirm the heterogeneous nature of tumor cells lineage. In this study, we set out to determine potential genetic markers associated with glioblastoma aggressiveness using multimodal deep learning.
Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-Agent LLMs
Alam, Firoj, Biswas, Md. Rafiul, Shah, Uzair, Zaghouani, Wajdi, Mikros, Georgios
In the past decade, social media platforms have been used for information dissemination and consumption. While a major portion of the content is posted to promote citizen journalism and public awareness, some content is posted to mislead users. Among different content types such as text, images, and videos, memes (text overlaid on images) are particularly prevalent and can serve as powerful vehicles for propaganda, hate, and humor. In the current literature, there have been efforts to individually detect such content in memes. However, the study of their intersection is very limited. In this study, we explore the intersection between propaganda and hate in memes using a multi-agent LLM-based approach. We extend the propagandistic meme dataset with coarse and fine-grained hate labels. Our finding suggests that there is an association between propaganda and hate in memes. We provide detailed experimental results that can serve as a baseline for future studies. We will make the experimental resources publicly available to the community.
Multimodal Analysis of White Blood Cell Differentiation in Acute Myeloid Leukemia Patients using a \beta-Variational Autoencoder
Mert, Gizem, Sadafi, Ario, Salehi, Raheleh, Navab, Nassir, Marr, Carsten
Biomedical imaging and RNA sequencing with single-cell resolution improves our understanding of white blood cell diseases like leukemia. By combining morphological and transcriptomic data, we can gain insights into cellular functions and trajectoriess involved in blood cell differentiation. However, existing methodologies struggle with integrating morphological and transcriptomic data, leaving a significant research gap in comprehensively understanding the dynamics of cell differentiation. Here, we introduce an unsupervised method that explores and reconstructs these two modalities and uncovers the relationship between different subtypes of white blood cells from human peripheral blood smears in terms of morphology and their corresponding transcriptome. Our method is based on a beta-variational autoencoder ({\ss}-VAE) with a customized loss function, incorporating a R-CNN architecture to distinguish single-cell from background and to minimize any interference from artifacts. This implementation of {\ss}-VAE shows good reconstruction capability along with continuous latent embeddings, while maintaining clear differentiation between single-cell classes. Our novel approach is especially helpful to uncover the correlation of two latent features in complex biological processes such as formation of granules in the cell (granulopoiesis) with gene expression patterns. It thus provides a unique tool to improve the understanding of white blood cell maturation for biomedicine and diagnostics.
Cross-Modality Translation with Generative Adversarial Networks to Unveil Alzheimer's Disease Biomarkers
Hassanzadeh, Reihaneh, Abrol, Anees, Hassanzadeh, Hamid Reza, Calhoun, Vince D.
Generative approaches for cross-modality transformation have recently gained significant attention in neuroimaging. While most previous work has focused on case-control data, the application of generative models to disorder-specific datasets and their ability to preserve diagnostic patterns remain relatively unexplored. Hence, in this study, we investigated the use of a generative adversarial network (GAN) in the context of Alzheimer's disease (AD) to generate functional network connectivity (FNC) and T1-weighted structural magnetic resonance imaging data from each other. We employed a cycle-GAN to synthesize data in an unpaired data transition and enhanced the transition by integrating weak supervision in cases where paired data were available. Our findings revealed that our model could offer remarkable capability, achieving a structural similarity index measure (SSIM) of $0.89 \pm 0.003$ for T1s and a correlation of $0.71 \pm 0.004$ for FNCs. Moreover, our qualitative analysis revealed similar patterns between generated and actual data when comparing AD to cognitively normal (CN) individuals. In particular, we observed significantly increased functional connectivity in cerebellar-sensory motor and cerebellar-visual networks and reduced connectivity in cerebellar-subcortical, auditory-sensory motor, sensory motor-visual, and cerebellar-cognitive control networks. Additionally, the T1 images generated by our model showed a similar pattern of atrophy in the hippocampal and other temporal regions of Alzheimer's patients.
Multimodal Analysis of memes for sentiment extraction
Alluri, Nayan Varma, Krishna, Neeli Dheeraj
Memes are one of the most ubiquitous forms of social media communication. The study and processing of memes, which are intrinsically multimedia, is a popular topic right now. The study presented in this research is based on the Memotion dataset, which involves categorising memes based on irony, comedy, motivation, and overall-sentiment. Three separate innovative transformer-based techniques have been developed, and their outcomes have been thoroughly reviewed.The best algorithm achieved a macro F1 score of 0.633 for humour classification, 0.55 for motivation classification, 0.61 for sarcasm classification, and 0.575 for overall sentiment of the meme out of all our techniques.
SEMEN ANALYSIS - Machine learning in the prediction of sperm motility
Automatic analysis of different types of clinical data is currently advancing rapidly, in particular, multimodal image analysis (learning simultaneously from various sources of data). At this year's ESHRE Annual Meeting, for example, there were several presentations on the subject of machine learning (a subfield of artificial intelligence) and reproductive outcomes. Though promising, most of such current research in human reproduction is, from a machine learning point of view, still in its infancy. Now, a new study from our group in Oslo shows that advanced machine learning methods for analysing videos of semen samples may be a useful tool in the investigation of male infertility.(1) Manual semen analysis is central to male infertility investigation, but is time-consuming and requires extensive training to obtain reproducible results.