Paper recorded by Signals 4 on 2026-10-02 in cs.CL. Abstract reproduced from arXiv; link to the original below.
Published 2026-10-02 on arXiv · recorded by Signals 4 on 2026-10-05
Category: cs.CL · 自然语言处理 · first seen 2026-10-05
Distillation has become a core primitive of large language model training, but its properties are not yet well understood. We take an entropic perspective, studying how the entropy of the student depends on the data and the divergence that define the distillation objective. We prove that forward KL inflates the entropy of the student above that of the teacher. Since cross-entropy training is a spe