Paper recorded by Signals 4 on 2026-09-29 in cs.CL. Abstract reproduced from arXiv; link to the original below.
Published 2026-09-29 on arXiv · recorded by Signals 4 on 2026-09-30
Category: cs.CL · 自然语言处理 · first seen 2026-09-30
Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for information to flow downward across generation steps is the decoded token. This narrow channel forces models to recompute intermediate results and to discard alternative continuations. In this work, we remove this bottleneck during pretraining, introducing