Large language models (LLMs) have become essential digital task assistance tools. Their training relies heavily on the collection of vast amounts of data, which may include copyright-protected or sensitive information. Recent studies on the detection of pretraining data in LLMs have primarily focused on sentence-level or paragraph-level membership inference attacks (MIAs), usually involving probability analysis of the target model prediction tokens. However, the proposed methods often demonstrate poor performance, specifically in terms of accuracy, failing to account for the semantic importance of textual content and word significance. To address these shortcomings, we propose Tag&Tab, a novel approach for detecting data that has been used as part of the LLM pretraining. Our method leverages advanced natural language processing (NLP) techniques to tag keywords in the input text - a process we term Tagging. Then, the LLM is used to obtain the probabilities of these keywords and calculate their average log-likelihood to determine input text membership, a process we refer to as Tabbing. Our experiments on three benchmark datasets (BookMIA, MIMIR, and the Pile) and several open-source LLMs of varying sizes demonstrate an average increase in the AUC scores ranging from 4.1% to 12.1% over state-of-the-art methods. Tag&Tab not only sets a new standard for data leakage detection in LLMs, but its outstanding performance is a testament to the importance of words in MIAs on LLMs.

Generative artificial intelligence (GenAI) models are a type of deep learning neural network model capable of learning from large datasets and generating new content from a given context. They represent a significant leap in the ability of the artificial intelligence (AI) field to not just interpret data but also to create something new, including text, images, videos, code, and sound [2]. Large language models (LLMs) are a type of GenAI model designed to understand and generate natural language. The market for LLMs is estimated to reach 40.8 billion USD by 2029, up from 10.5 billion USD in 2022 [10]. Organizations are currently competing to develop the most sophisticated LLM capable of mimicking human-like conversations and tasks. This has led to the creation of models such as OpenAI's ChatGPT,