How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure

Paper recorded by Signals 4 on 2026-09-24 in cs.CL. Abstract reproduced from arXiv; link to the original below.

Published 2026-09-24 on arXiv · recorded by Signals 4 on 2026-09-25

Category: cs.CL · 自然语言处理 · first seen 2026-09-25

Abstract

Evaluations of LLM systems routinely average over small prompt sets and report models as a ranked table. We ask how much confidence such a table deserves, using LLM-based prompt-structure inference as the case study: eight open model variants across five families and 8B to 675B parameters, caching disabled, 293 raw intermediate representations persisted. The measured phenomenon is unstable to begi

Read on arXiv →