Paper recorded by Signals 4 on 2026-10-22 in cs.LG. Abstract reproduced from arXiv; link to the original below.
Published 2026-10-22 on arXiv · recorded by Signals 4 on 2026-10-23
Category: cs.LG · 机器学习 · first seen 2026-10-23
Standard implementations of softmax, log-softmax, and cross-entropy have two numerical bugs. The first bug, Softmax Collapse, rounds the target logit's gradient to zero if that one logit is much larger than the others, resulting in gradients that no longer sum to zero. The second bug, Softmax Cancellation, occurs when student logits are close to teacher logits, and produces non-zero gradients even