Paper recorded by Signals 4 on 2026-09-02 in cs.AI. Abstract reproduced from arXiv; link to the original below.
Published 2026-09-02 on arXiv · recorded by Signals 4 on 2026-09-03
Category: cs.AI · 人工智能 · first seen 2026-09-03
Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy