Large Language Model
A Implementation Details 1 A.1 Preliminary Study 2 The basic GPT-2 model
Model details can be found at https://huggingface.co/gpt2 Popular phrases such as "pair of" are examples of repetitive
Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective Huayang Li Tian Lan Zihao Fu Deng Cai Lemao Liu Nigel Collier
In this work, we aim to advance our understanding by presenting a straightforward and fundamental explanation from the data perspective. Our preliminary investigation reveals a strong correlation between the degeneration issue and the presence of repetitions in training data. Subsequent experiments also demonstrate that by selectively dropping out the attention to repetitive words in training data, degeneration can be significantly minimized.
From Causal to Concept-Based Representation Learning
A key question in this growing field is to provide rigorous conditions under which latent factors can be identified and thus, potentially learned.
Training Chain-of-Thought via Latent-Variable Inference Du Phan Matthew D. Hoffman
One can also improve LLMs' performance on a specific task by
Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models
In this work, we present MuDI, a novel framework that enables multi-subject personalization by effectively decoupling identities from multiple subjects.