LlamaIndex ExtractBench scores document extraction with strict grounding at IoU 0.5
LlamaIndex has outlined how ExtractBench grades document extraction systems on provenance, not just extracted text. The benchmark treats a field as correct only when both the value and its word-level citation box meet IoU 0.5, penalizing answers that look right but cite the wrong span. Related posts describe the suite more broadly: 14 systems evaluated across 370 enterprise documents, with stress tests on degraded scans, historical filings, and long-list completeness where missing rows can hide behind valid schema output. For teams shipping document AI into audits, compliance, or operations workflows, the signal is that vendor accuracy numbers may overstate trustworthiness if grounding and completeness are not measured separately. The packet does not publish full leaderboard scores or independent replication.
LlamaIndex ExtractBench scores document extraction with strict grounding at IoU 0.5
Most document extraction APIs can't tell you where a value came from. LlamaIndex: "For ExtractBench, we scored grounding strictly: a field only counts if the value AND its citation are correct, word-level box at IoU 0.5."
Key takeaway
ExtractBench scores document extraction on joint value-and-citation correctness at IoU 0.5, so plausible fields with wrong provenance no longer pass.
What happened
In an August 17 post on X, LlamaIndex said most document extraction APIs cannot show where a value came from and described ExtractBench grounding rules: a field counts only if the extracted value and its citation are both correct, using a word-level box at IoU 0.5.
Earlier ExtractBench posts on X reported testing 14 extraction systems on 370 enterprise documents, including non-digital 1950s regulatory filings and hand-filled tax forms, and highlighted long-list completeness as a failure mode where missing rows can look structurally valid.
Evidence
ExtractBench requires both the extracted value and its citation to be correct at IoU 0.5.
LlamaIndex (X) · attributed
For ExtractBench, we scored grounding strictly: a field only counts if the value AND its citation are correct, word-level box at IoU 0.5.
ExtractBench tested 14 systems on non-digital and degraded documents.
LlamaIndex (X) · attributed
For ExtractBench, we tested 14 systems on documents that weren't born digital: 1950s regulatory filings, hand-filled tax forms, and pages deg
ExtractBench evaluated 370 enterprise documents across 14 systems with a long-list completeness focus.
LlamaIndex (X) · attributed
We released ExtractBench yesterday: 370 enterprise docs, 14 systems. The hardest test: long-list c
A critical failure mode is a missing row that appears structurally correct.
LlamaIndex (X) · attributed
The most dangerous document extraction failure isn't a wrong value. It's a missing row that looks like nothing is wrong.
Why it matters
Production document AI pipelines need benchmarks that catch silent omissions and bad citations, not headline accuracy alone.
Limits and uncertainties
The packet provides benchmark framing and test design but does not include full per-system scores or third-party replication.
Several source excerpts in the packet are truncated and do not show complete benchmark results.
Practical implications
Before selecting a document extraction vendor, require evaluation on grounding at IoU 0.5, not value accuracy alone.
Validate OCR-heavy and legacy-document workloads separately from clean digital PDF benchmarks.
Add long-list completeness checks to catch silent row-drop failures in production schemas.
What to watch
Whether LlamaIndex publishes full ExtractBench leaderboard results and methodology details beyond the X posts.
How extraction vendors respond with citation-grounding and completeness metrics aligned to ExtractBench criteria.
Independent replication on enterprise legacy documents and long-list extraction tasks.
Original reporting: Most document extraction APIs can't tell you where a value came from. For ExtractBench, we scored grounding strictly: a field only counts if the value AND its citation are correct…
