Keep It Light! Simplifying Image Clustering Via Text-Free Adapters