This paper introduces RETSim (Resilient and Efficient Text Similarity), a lightweight, multilingual deep learning model trained to produce robust metric embeddings for near-duplicate text retrieval, clustering, and dataset deduplication tasks. We demonstrate that RETSim is significantly more robust and accurate than MinHash and neural text embeddings, achieving new state-of-the-art performance on dataset deduplication, adversarial text retrieval benchmarks, and spam clustering tasks. We also introduce the W4NT3D benchmark (Wiki-40B 4dversarial Near-T3xt Dataset) for evaluating multilingual, near-duplicate text retrieval capabilities under adversarial settings. RETSim and the W4NT3D benchmark are open-sourced under the MIT License at https://github.com/google/unisim.

In contrast, transformers employ a "Scaled Dot-Product Attention" mechanism that is parallelizable. This unique attention mechanism allows for large-scale pretraining. Additionally, self-supervised pretraining paradigm such as masked language modeling onlarge unlabeled datasets enabled transformers to be trained without costly annotations. Transformer model, although originally designed for the NLP [3] domain, Transformers have witnessed adaptations in various domains such as computer vision [5, 6], remote sensing [7], time series [8], speech processing [9] and multimodal learning [10]. Consequently, modality specific surveys emerged, focusing on medical imaging [11-13] and biomedical language models [14] in the medical domain. This paper aims to provide comprehensive overview of Transformer models utilized across multiple modalities of data to address healthcare objectives. We discuss pre-training strategies to manage the lack of robust and annotated healthcare datasets. The rest of the paper is organized as follows: Section 2 discusses the strategy to search for relevant citations; Section 3 describes the architecture of the original transformer; Section 4 describes the two primary Transformer variants: the Bidirectional Encoder Representations from Transformers (BERT) and the Vision Transformer (ViT). Section 5 describes advancements in large language models (LLM), and section 6 through 12 provides a review of Transformers in healthcare.