Attention mechanisms are central to the success of large language models (LLMs), enabling them to capture intricate token dependencies and implicitly assign importance to each token. Recent studies have revealed the sink token, which receives disproportionately high attention despite their limited semantic role. In this paper, we first expand the relationship between the sink token and other tokens, moving beyond attention to explore their similarity in hidden states, considering the layer depth. We observe that as the layers get deeper, the cosine similarity between the normalized hidden states of the sink token and those of other tokens increases, and that the normalized hidden states of the sink token exhibit negligible changes. These imply that other tokens consistently are directed toward the sink token throughout the layers. Next, we propose a dynamic token selection method, called OrthoRank, using these findings to select important tokens. Specifically, in a certain layer, we define token importance by the speed at which the token moves toward the sink token. This is converted into orthogonality with the sink token, meaning that tokens that are more orthogonal to the sink token are assigned greater importance. Finally, through extensive experiments, we demonstrated that our method results in lower perplexity and higher zero-shot accuracy compared to layer pruning methods at the same sparsity ratio with comparable throughput, while also achieving superior performance on LongBench.

Photos are taken by a regular smartphone.

It is a Monday afternoon in August, and I am on the internet watching a former cable-news anchor interview a dead teenager on Substack. This dead teenager--Joaquin Oliver, killed in the mass shooting at Marjory Stoneman Douglas High School, in Parkland, Florida--has been reanimated by generative AI, his voice and dialogue modeled on snippets of his writing and home-video footage. The animations are stiff, the model's speaking cadence is too fast, and in two instances, when it is trying to convey excitement, its pitch rises rapidly, producing a digital shriek. How many people, I wonder, had to agree that this was a good idea to get us to this moment? I feel like I'm losing my mind watching it. Jim Acosta, the former CNN personality who's conducting the interview, appears fully bought-in to the premise, adding to the surreality: He's playing it straight, even though the interactions are so bizarre. Acosta asks simple questions about Oliver's interests and how the teenager died.

Few fantasies are harder to wipe away than the romance of a clean slate. Every January, when we're twitchy with regret and self-loathing, advertisers blare, "New Year, new you," urging us to jettison our failures and start fresh. In fiction, self-reinvention is a perennial theme, often shadowed by the suspicion that it can't be done. Lately, novelists have put a political spin on the idea, counterposing hopeful acts of individual self-fashioning to the immovable weight of circumstance. Halle Butler's "The New Me" (2019), a millennial office satire, finds its temp heroine, Millie, trying to life-hack her way out of loneliness and professional drift--buy a plant, whiten her teeth, make friends, think positive.

When taking photos under an environment with insufficient light, the exposure time and the sensor gain usually require to be carefully chosen to obtain images with satisfying visual quality.

During image editing, existing deep generative models tend to re-synthesize the entire output from scratch, including the unedited regions. This leads to a significant waste of computation, especially for minor editing operations. In this work, we present Spatially Sparse Inference (SSI), a general-purpose technique that selectively performs computation for edited regions and accelerates various generative models, including both conditional GANs and diffusion models.