[D] K-means on Neural Network Layers (Text data) • r/MachineLearning

I have a model that's trained to detect various concepts on biomedical papers. I'm using the last dense layer before the output to use as vector embeddings for these medical papers. I then use k-means on the embedding vectors to cluster the papers together. I prefer this over a pure unsupervised approach as these embeddings will be forced to focus on biological similarity over biological & semantic. I have about 10M papers and each cluster needs to have 50 elements.