It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention

Paper recorded by Signals 4 on 2026-09-08 in cs.CL. Abstract reproduced from arXiv; link to the original below.

Published 2026-09-08 on arXiv · recorded by Signals 4 on 2026-09-09

Category: cs.CL · 自然语言处理 · first seen 2026-09-09

Abstract

Large Language Models (LLMs) often exhibit "Attention Sink" (AS) and the accompanying "Massive Activations" (MAs) at the initial position of a sequence. These phenomena frequently co-occur, and MAs can pose challenges for low-bit quantization. In this study, we analyze the factors underlying AS and MAs that emerge at the initial position regardless of the token occupying it. Our experiments sugges

Read on arXiv →