What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models

Paper recorded by Signals 4 on 2026-10-27 in cs.AI. Abstract reproduced from arXiv; link to the original below.

Published 2026-10-27 on arXiv · recorded by Signals 4 on 2026-10-28

Category: cs.AI · 人工智能 · first seen 2026-10-28

Abstract

Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal va

Read on arXiv →