Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data

Paper recorded by Signals 4 on 2026-09-16 in cs.LG. Abstract reproduced from arXiv; link to the original below.

Published 2026-09-16 on arXiv · recorded by Signals 4 on 2026-09-17

Category: cs.LG · 机器学习 · first seen 2026-09-17

Abstract

Large Language Models (LLMs) are now routinely trained using synthetic data, since high-quality human data has been exhausted by the ever increasing needs of larger and larger models. However, recursive training on synthetic data frequently induces model collapse, a degenerative feedback loop where models progressively forget the true underlying data distribution. Training on a mixture of syntheti

Read on arXiv →