Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining

Paper recorded by Signals 4 on 2026-08-28 in cs.LG. Abstract reproduced from arXiv; link to the original below.

Published 2026-08-28 on arXiv · recorded by Signals 4 on 2026-08-31

Category: cs.LG · 机器学习 · first seen 2026-08-31

Abstract

Pretraining accounts for a large fraction of the total computational cost in LLM training. However, noise-dominant gradients and the highly ill-conditioned loss landscape bring severe challenges. Although modern adaptive optimizers such as AdamW and Muon have achieved great success in large-scale pretraining, their reliance on gradient normalization offers limited mitigation of the ill-conditioned

Read on arXiv →