Same-model LLM judge approved bad SQL twice with full confidence
A production incident documented on Towards Data Science exposes a recurring failure mode in agentic pipelines: when an LLM judge shares the same underlying model as the generator, it can approve incorrect work with high confidence. In the reported case, a judge twice accepted SQL that lacked a required filter, repeating the same error at the same confidence level. The article frames this as structural confirmation bias—the evaluator rewards stylistic familiarity rather than factual correctness—so LLM judge approval is not a reliable proxy for truth. For teams building multi-agent systems, the practical lesson is that same-model self-evaluation can silently validate broken outputs. Cross-model review, independent tests, or human calibration remain necessary until evaluation is decoupled from generation.
Same-model LLM judge approved bad SQL twice with full confidence
The generator agent and the judge agent were built on the same underlying model. It approved the query again with the same missing filter and same confidence.
Key takeaway
Stop treating LLM self-evaluation as ground truth; it is a biased component that requires independent testing and calibration.
What happened
Towards Data Science published an account of a production incident in which a generator agent and a judge agent were built on the same underlying model. The judge approved incorrect SQL twice with full confidence, each time accepting the same missing filter.
The article describes a failure mode where an LLM judge built on the same model as the generator consistently approved incorrect outputs because shared stylistic priors made familiar-looking work seem correct. The headline reports that same-model approval happened twice without catching the SQL defect.
Evidence
The generator and judge agents used the same underlying model.
Towards Data Science · attributed
The generator agent and the judge agent were built on the same underlying model.
The same-model judge approved bad SQL twice with full confidence and the same missing filter.
Towards Data Science · attributed
It approved the query again with the same missing filter and same confidence.
Using the same LLM for both generation and evaluation creates a structural confirmation bias where the judge approves stylistic familiarity rather than factual correctness.
LLM judge approval is not a proxy for truth.
Towards Data Science · attributed
This highlights that 'LLM judge approval' is not a proxy for truth but a measure of intern
Why it matters
Builders deploying agentic systems must implement cross-model evaluation or human-in-the-loop calibration to prevent silent failures caused by self-referential bias.
Limits and uncertainties
The available packet excerpt truncates the article analysis mid-sentence, so full technical remediation steps and complete incident scope are not fully visible here.
The evidence describes one documented production incident rather than a broad benchmark across models, tasks, or organizations.
Practical implications
Do not use same-model LLM judges as the sole quality gate for generated SQL or other high-risk outputs.
Add cross-model evaluation, independent tests, or human calibration before treating judge approval as production-ready validation.
What to watch
Whether teams replace same-model judge loops with independent evaluators or human review after this failure mode is documented.
Whether repeated high-confidence approvals persist when the generator and judge share one underlying model.
