Paper recorded by Signals 4 on 2026-09-18 in cs.LG. Abstract reproduced from arXiv; link to the original below.
Published 2026-09-18 on arXiv · recorded by Signals 4 on 2026-09-21
Category: cs.LG · 机器学习 · first seen 2026-09-21
Gating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why. We argue and provide experimental evidence that such gates supply two different things that softmax attention lacks: abstention and noise filtering. The first is abstention, which allows an attention head to output nothing, bypassing the requirement that attention weights must s