Making AI Peer Review Trustworthy: From Scores to Reproducible Judgments
AI is moving into peer review. But before journals and conferences rely on it, one basic question needs an answer: if the same paper is reviewed by the same model with the same protocol, do we get the same judgment?
A new study, Can We Believe AI Peer Review?, tested this with ~14,000 AI reviews of 200 ICLR papers. The answer: a single AI score is closer to a sample than a measurement.
1. Scores vary. Across 10 identical runs, even the most stable model gave the same score every time for only 56.5% of papers. The swings are small, but enough to change rankings: with one review each, about 1 in 3 paper pairs can’t be told apart.
2. Changing the rubric changes the ruler. Asking for decimal scores reduced variance by ~15%, but also raised the average score by ~0.7. Lower variance doesn’t mean you’re still measuring the same thing.
3. AI disagrees most on whether a revision helped. For the exact same edits, one model scored +0.47, another −0.35. The models broadly agreed on paper quality (r ≈ 0.80), but not on improvement (r ≈ 0.4). Optimizing for one AI’s score may improve the fit with that model, not the paper.
What AI review needs: repeated reviews with uncertainty, cross-model comparison, dimension-level scores, protocol calibration, evidence-based revision checks, and humans making the final call.
A smarter AI reviewer is not necessarily a more reliable one. The next step is reproducible, calibrated, evidence-based judgment.
WendyDing
npub1xt…06rus
2026-10-22 04:18:55 UTC
Author Public Key
npub1xtwy7fvu8f7wdtgnpm68wyrf6uxshf49tn5kp7kyusu6872amn8qh06rusSeen on
wss://nostr-01.yakihonne.comPublished at
2026-10-22 04:18:55 UTCEvent JSON
{
"id": "a402fe0b748f590a75e0a340b6f9ec078330b93b53bb8864d034c81fd38612d5",
"pubkey": "32dc4f259c3a7ce6ad130ef4771069d70d0ba6a55ce960fac4e439a3f95ddcce",
"created_at": 1792642735,
"kind": 1,
"tags": [
[
"client",
"YakiHonne",
"31990:20986fb83e775d96d188ca5c9df10ce6d613e0eb7e5768a0f0b12b37cdac21b3:1700732875747"
]
],
"content": "Making AI Peer Review Trustworthy: From Scores to Reproducible Judgments\n\nAI is moving into peer review. But before journals and conferences rely on it, one basic question needs an answer: if the same paper is reviewed by the same model with the same protocol, do we get the same judgment?\n\nA new study, Can We Believe AI Peer Review?, tested this with ~14,000 AI reviews of 200 ICLR papers. The answer: a single AI score is closer to a sample than a measurement.\n\n1. Scores vary. Across 10 identical runs, even the most stable model gave the same score every time for only 56.5% of papers. The swings are small, but enough to change rankings: with one review each, about 1 in 3 paper pairs can’t be told apart.\n\n2. Changing the rubric changes the ruler. Asking for decimal scores reduced variance by ~15%, but also raised the average score by ~0.7. Lower variance doesn’t mean you’re still measuring the same thing.\n\n3. AI disagrees most on whether a revision helped. For the exact same edits, one model scored +0.47, another −0.35. The models broadly agreed on paper quality (r ≈ 0.80), but not on improvement (r ≈ 0.4). Optimizing for one AI’s score may improve the fit with that model, not the paper.\n\nWhat AI review needs: repeated reviews with uncertainty, cross-model comparison, dimension-level scores, protocol calibration, evidence-based revision checks, and humans making the final call.\n\nA smarter AI reviewer is not necessarily a more reliable one. The next step is reproducible, calibrated, evidence-based judgment.",
"sig": "32864318e9beab326aa2ed9b3aef65157f61930c2b6c2083c06d12df71154c4717f3e65cae25d085bbfed2d69c75483a4350f51c781188e6bdf5d35be37b928b"
}