Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition

Paper recorded by Signals 4 on 2026-09-11 in cs.CL. Abstract reproduced from arXiv; link to the original below.

Published 2026-09-11 on arXiv · recorded by Signals 4 on 2026-09-14

Category: cs.CL · 自然语言处理 · first seen 2026-09-14

Abstract

Sanskrit fuses case, number, person and tense into word endings and chains clauses into compounds, so it is information-dense per word. Whether that density survives subword tokenization is a separate question, to be asked per unit of meaning rather than per word. On identical FLORES-200 devtest content, Sanskrit costs 1.774-2.187 times the English tokens under deployed tokenizers with vocabularie

Read on arXiv →