Amazon Researchers Find LLM Judge Panels Overstate Agreement

Two Amazon researchers published a paper Aug. 26 showing that panels of AI models used to judge other AI models often agree for the wrong reason. Krishna Balasubramanian and Sasha Podkopaev found that shared training data, prompts or model families can make judges look independent when they are not.

Balasubramanian is an Amazon Scholar and a professor at the University of California, Davis; Podkopaev is an applied scientist at AWS. Their paper, done with Shiva Prasad Kasiviswanathan, was accepted to the International Conference on Machine Learning, or ICML, in 2026, and first posted to arXiv on Jan. 29, 2026.

Why agreement can be a shared blind spot, not evidence

The industry practice the paper targets is using a panel of large language models to grade another model's answers, a cheaper substitute for human reviewers. Standard aggregation methods, including Dawid-Skene and weighted majority voting, assume each judge errs independently of the others. Balasubramanian and Podkopaev said that assumption breaks when judges share an architecture, a training set or a prompt template, because they then tend to make the same mistake at the same time, which looks like consensus.

An Ising model, borrowed from physics, scores the difference

Their method treats a panel of judges as a network and applies Ising models, a tool from statistical physics originally used to describe how neighboring atoms' magnetic spins influence each other, to estimate how correlated any two judges' errors are. The system needs no human-labeled data: it infers each judge's reliability and its correlation with the others from the judges' outputs alone.

Gains of 9 to 14 percent across three tasks

Tested on panels of 10 judges, the dependence-aware method beat the strongest baseline, a panel weighted by each judge's historical accuracy, by 9 to 14 percent, the paper said. The three tasks graded whether retrieved documents were relevant to a query, whether text was toxic, and whether a generated summary faithfully represented its source, three jobs the paper said are already outsourced to LLM panels at production scale.

Task	Weighted majority vote	Dependence-aware method
Relevance classification	0.820	0.912
Toxicity detection	0.694	0.792
Summarization assessment	0.737	0.806

The line the paper is built around

"When LLM judges agree, we should ask why," Balasubramanian and Podkopaev wrote in an Amazon Science blog post published alongside the paper. "Sometimes agreement is independent evidence. Sometimes it is a shared blind spot."

The paper's skepticism about self-reported agreement echoes a pattern across the industry this month: OpenAI's own ExploitBench and ExploitGym scores for GPT-6 Astra have no named independent runner, and neither do the benchmark claims behind Cognition's SWE-2 pricing pitch.

What's still untested

QuantumZeitgeist, an outlet covering physics-adjacent research, wrote Sept. 8 that the method "enables identification of redundant judges and systematic blind spots" without needing ground-truth labels to calibrate against. The paper tested three tasks with binary labels. Balasubramanian and Podkopaev have not said whether the method holds on tasks with more than two possible labels, or on panels larger than 10 judges.