Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models

Paper recorded by Signals 4 on 2026-10-01 in cs.CL. Abstract reproduced from arXiv; link to the original below.

Published 2026-10-01 on arXiv · recorded by Signals 4 on 2026-10-02

Category: cs.CL · 自然语言处理 · first seen 2026-10-02

Abstract

Keyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics. A 661.6M parameter model (approx. 65% code/technical text; no dedicated SFT) and a 1,109M model (web-heavy multi-phase curriculum; 6B-token tool-SFT) share decoder

Read on arXiv →