Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

Paper recorded by Signals 4 on 2026-09-01 in cs.CL. Abstract reproduced from arXiv; link to the original below.

Published 2026-09-01 on arXiv · recorded by Signals 4 on 2026-09-02

Category: cs.CL · 自然语言处理 · first seen 2026-09-02

Abstract

Logit-based knowledge distillation (KD) is used to train smaller language models (LMs) via supervision from stronger teachers, but whether its benefits are consistent across training stages remains unclear. Through controlled experiments, we find that forward Kullback-Leibler (KL) distillation--the standard KD formulation--with post-trained teachers behaves fundamentally differently during mid-tra

Read on arXiv →