Paper recorded by Signals 4 on 2026-10-08 in cs.CL. Abstract reproduced from arXiv; link to the original below.
Published 2026-10-08 on arXiv · recorded by Signals 4 on 2026-10-09
Category: cs.CL · 自然语言处理 · first seen 2026-10-09
Tokenizers are commonly optimized for compression, but a compact vocabulary does not necessarily distribute its capacity evenly across languages. We introduce the Latent Core Tokenizer (LCT), a language-agnostic approach that separates structural discovery from vocabulary construction. LCT uses Minimum Description Length, entropy-based boundary signals, and morphotactic constraints to identify reu