Paper recorded by Signals 4 on 2026-11-02 in cs.AI. Abstract reproduced from arXiv; link to the original below.
Published 2026-11-02 on arXiv · recorded by Signals 4 on 2026-11-03
Category: cs.AI · 人工智能 · first seen 2026-11-03
Most parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step. This memory grows with the number and size of the experts, so it rises as MoE models scale. We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away. It brings optimizer-in-bac