Paper recorded by Signals 4 on 2026-09-17 in cs.CL. Abstract reproduced from arXiv; link to the original below.
Published 2026-09-17 on arXiv · recorded by Signals 4 on 2026-09-18
Category: cs.CL · 自然语言处理 · first seen 2026-09-18
Reasoning and agentic workloads increasingly demand efficient long-context inference. Yet full-attention decoding reads the growing history at every step, regardless of its benefit to the next prediction. We show that a pretrained model's decoding states already contain information predictive of this benefit, before the global read. Building on this finding, we introduce On-Demand Attention (ODA),