Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks
Paper recorded by Signals 4 on 2026-11-02 in cs.CL. Abstract reproduced from arXiv; link to the original below.
Published 2026-11-02 on arXiv · recorded by Signals 4 on 2026-11-03
Category: cs.CL · 自然语言处理 · first seen 2026-11-03
Abstract
Every multimodal-LLM multiple-choice benchmark ends with a scoring rule that converts a model's response, often free-form text rather than a clean option letter, into a right/wrong verdict, buried in its code. Unlike the dataset, this rule is chosen without justification and left unexamined, yet it sets the published number. We reproduce the official scorers of eight peer-reviewed benchmarks on fi
Read on arXiv →
Cite this page: Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks: the #18 most recent of 628 cs.CL papers we have recorded (as of 2026-11-02). Source: Signals 4 (Signals API) — https://data.jiangzhang.ca/signals4/t/papers/same-outputs-different-scores-auditing-the-unchecked-graders-of-multimodal-llm-b.html
Free to quote with attribution to “Signals 4 (Signals API)”. Machine-readable:
papers.json
Get 4 AI signals a day by email — free.
Get 4 AI signals a day by email — free