SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models

Paper recorded by Signals 4 on 2026-10-02 in cs.CL. Abstract reproduced from arXiv; link to the original below.

Published 2026-10-02 on arXiv · recorded by Signals 4 on 2026-10-05

Category: cs.CL · 自然语言处理 · first seen 2026-10-05

Abstract

Large language models are increasingly used where small syntactic errors matter, yet character-level reasoning is still evaluated mostly through isolated probes and aggregate accuracy. We introduce SyntaxBench, a diagnostic benchmark and statistical evaluation framework for character-level reasoning. It contains five core tasks, character counting, letter containment, palindrome detection, edit di

Read on arXiv →