Diagnosing Model--Item Interactions in Language Model Benchmarks with LSIRM

Paper recorded by Signals 4 on 2026-11-02 in cs.CL. Abstract reproduced from arXiv; link to the original below.

Published 2026-11-02 on arXiv · recorded by Signals 4 on 2026-11-03

Category: cs.CL · 自然语言处理 · first seen 2026-11-03

Abstract

Language models with similar benchmark accuracy can succeed on different items. Overall scores do not show where their performance differs or which item properties might explain those differences. We demonstrate how the latent space item response model (LSIRM) can help evaluators investigate these patterns and their consequences for model comparison. We analyze 1,348 language models on 5,761 BBH i

Read on arXiv →