Research
Preprint proposes pairwise signals to tighten LLM math rankings
A revised preprint reports a method intended to make mathematical-reasoning evaluations of large language models more precise when benchmarks are small or model outputs vary between runs.
arXiv
What happened
The July 20, 2026 revision of “Evaluating LLMs When They Do Not Know the Answer” proposes combining ordinary accuracy results with model judgments that compare two auxiliary reasoning chains.
The authors treat those comparisons as auxiliary signals in a statistical estimator. Rather than relying only on whether a final answer is correct, the method uses judgments about which of two candidate solutions is better.
In author-run simulations and tests on GPQA Diamond, AIME 2025, and GSM8K, the paper reports more precise performance estimates and generally more reliable model rankings than naive sample averaging. The reported gains were strongest with small evaluation samples.
AIME 2025 has 30 problems, so changing one item moves raw accuracy by more than three percentage points. That makes small score differences especially coarse on that benchmark.
Why it matters
Benchmark rankings often guide claims about which models are stronger. Separate research has found uncertainty in benchmark scores and substantial ranking variability, but it did not test this paper’s method. More precise estimates of a fixed score also do not show that a benchmark measures reasoning validity.
What remains unclear
- The findings are self-reported in an unreviewed preprint, with no independent replication or peer-reviewed validation identified in the dossier.
- The experiments cover selected math and science benchmarks and configurations, not all models, benchmarks, or non-mathematical tasks.
- It remains unclear when model-produced pairwise judgments provide useful signal and when they may introduce errors or bias.
Sources
-
Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals
Used for: The revised preprint, its proposed estimator, benchmark scope, and reported results.
-
Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores
Used for: Context on uncertainty in LLM benchmark scores.
-
Quantifying Ranking Uncertainty in LLM Benchmarks
Used for: Context on ranking variability in LLM benchmarks.