Machine learning methods fail to provide cohesive atheoretical construction of personality traits from semantic embeddings

Here, we test this hypothesis using novel machine learning methods to create a bottom-up, atheoretical model of personality from the same trait-descriptive adjective list that led to the dominant, contemporary model of personality (the Big Five). We then compare the descriptive utility of this machine learning method (resulting in lexical clusters) by comparing it to the established Big Five personality model in how well these describe conversations online (on Reddit forums). Our analysis of 1 million online comments shows that the Big Five model provides a much more powerful and interpretable description of these communities and the differences between them. Specifically, the dimensions of Agreeableness, Conscientiousness, and Neuroticism effectively distinguish Reddit communities. In contrast, our lexical clusters do not provide meaningful distinctions and fail to describe the spread. Validation against the International Personality Item Pool confirmed the Big Five model's superior psychometric coherence, and our machine learning methods notably failed to recover the trait of Extraversion. These results affirm the robustness of the Big Five, while also showing that the semantic structure of personality is likely depending on social context. Our findings suggest that while machine learning can help with understanding and explaining human behavior, especially by checking ecological validity of existing theories, machine learning methods may not be able to replace established psychological theories.