Paper recorded by Signals 4 on 2026-10-27 in cs.AI. Abstract reproduced from arXiv; link to the original below.
Published 2026-10-27 on arXiv · recorded by Signals 4 on 2026-10-28
Category: cs.AI · 人工智能 · first seen 2026-10-28
Weight decay and explicit weight-norm constraints improve optimizer performance in language model pretraining, yet a unified understanding of why such control helps and how much is needed remains incomplete. We analyze norm control in Muon over the entire training trajectory using the \emph{accumulated effective learning rate}, $C_T=\sum_{t<T}η_t/\|W_t\|_F$, where $η_t$ is the nominal learning rat