Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
Paper recorded by Signals 4 on 2026-09-03 in cs.AI. Abstract reproduced from arXiv; link to the original below.
Published 2026-09-03 on arXiv · recorded by Signals 4 on 2026-09-04
Category: cs.AI · 人工智能 · first seen 2026-09-04
Abstract
Language-model judges now gate training data, score generations, and drive leaderboards. The judge is then a measurement instrument, resting on one rarely stated assumption: the same request, sent to the same model name, reads the same tomorrow. We audited that assumption in two preregistered campaigns with every threshold fixed in advance; neither got past validating its instrument. Across 52,988
Read on arXiv →
Cite this page: Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints: the #182 most recent of 300 cs.AI papers we have recorded (as of 2026-09-03). Source: Signals 4 (Signals API) — https://data.jiangzhang.ca/signals4/t/papers/clean-engineering-unstable-measurement-a-preregistered-reliability-failure-of-bl.html
Free to quote with attribution to “Signals 4 (Signals API)”. Machine-readable:
papers.json
Get 4 AI signals a day by email — free.
Get 4 AI signals a day by email — free