Adaptive Frequency Threshold Byte Pair Encoding for Mixed Natural Language and Structured Symbolic Corpora

Paper recorded by Signals 4 on 2026-10-23 in cs.LG. Abstract reproduced from arXiv; link to the original below.

Published 2026-10-23 on arXiv · recorded by Signals 4 on 2026-10-26

Category: cs.LG · 机器学习 · first seen 2026-10-26

Abstract

Modern language models rely on subwording tokenization to define the discrete units of input processing, typically using methods like Byte Pair Encoding (BPE) that treat whitespace as a fixed semantic boundary. This design is inherently limited, as many meaningful concepts---including multiword expressions and the structured environments of formal languages like LaTeX---frequently span across whit

Read on arXiv →