- Source
- arXiv
- Published
- Runtime
- 0:00
- Snippets
- 7
A conversation between
Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays
§02
Snippets
-
The natural procedure for per-field accept/review silently violates the error-rate control contract on real documents, failing in 95% of validation splits.
Trust in automated document extraction depends on honest error guarantees; naive methods create false confidence without delivering promised safety.
-
Score-refit leakage—tuning thresholds on data then evaluating on the same data—causes coverage to collapse (0.416) while inflating claimed risk (0.127 vs. budget 0.10).
A common workflow silently breaks risk guarantees; proper train/validation splits are mandatory, not optional.
-
Tie-mass pathology—where many fields share identical scores—collapses the threshold search grid from 0.030 to 0.001, breaking calibration.
Real model outputs contain ties; unhandled ties silently destroy the tuning procedure's ability to find valid thresholds.
-
Document clustering inflates effective sample size; design effects of 1.84–2.45 mean standard binomial error bars are too tight by a factor of 2.
Treating documents as independent undercounts uncertainty; clustering-aware methods are necessary for honest error bounds.
-
Mondrian Learn-then-Test with exact binomial tails yields PAC certificates; cluster-corrected method achieves 0.140 coverage at risk 0.068—the only tier matching document structure.
Different use cases need different guarantees; formal certificates cost coverage but deliver honest error promises where it matters most.
-
Support-bin (taxonomy-based) dominates sonnet-5 (p<1e-4) but does not replicate on haiku or qwen; conditioning helps exactly where pooled methods lack guarantees.
No single strategy works everywhere; effectiveness depends on model accuracy and whether certification is required.
-
Blind three-annotator human audit confirms the practical tier's accepted-set risk at 1.3%—well within its 10% budget (Fleiss' kappa=0.83, pessimistic labeling bias).
Lab guarantees mean nothing without real-world verification; human gold-standard audit proves the method's claims translate to actual field accuracy.
§03
Synthesis
The Core Problem: Document Extraction Can't Guarantee Error Control
When a document-extraction system decides to accept or review a field, it needs a guarantee that accepted fields won't exceed some target error rate (alpha). This is called selective risk control — accept only when confident enough. The authors show that the natural baseline approach silently violates this guarantee on real documents. They diagnose why, build a validity ladder of fixes with different rigor levels, and identify when conditioning on document structure actually helps.
Three Failure Modes
The authors tested on 13,859 real fields from 800 CORD receipts processed by Claude Sonnet 3.5 (49% correct). Three systematic failures emerged:
Document clustering: Fields in the same document are correlated, not independent. This inflates the effective sample size via a design effect of 1.84–2.45, breaking assumptions of standard control procedures.
Score-refit leakage: When you train a threshold on the same data used to validate it, the threshold optimizes for that split. Result: the method reports 41.6% coverage at 12.7% risk, violating the nominal 10% budget in 95% of random resplits. Real risk is higher than claimed.
Tie-mass pathology: Many fields receive identical scores (ties). This collapses the threshold grid from 0.030 to 0.001 unique values, making it impossible to fine-tune acceptance decisions.
A Validity Ladder: Trading Coverage for Honesty
The authors propose four tiers, each with stronger guarantees:
Fit/Val split (weakest): Separate train and validation data. Recovers expected risk control (31.8% coverage at 9.6% realized risk for alpha=0.10), but this is an average — realized risk still exceeds alpha in 47.5% of random resplits. No per-instance certificate.
Learned fusion with Mondrian Learn-then-Test: Applies exact binomial confidence bounds. Field-level independence: 17.1% coverage, 6.8% risk. Adds a certificate but narrow coverage.
Cluster correction: Accounts for document-level clustering. Coverage drops to 14.0%, risk 6.8% — acknowledges real structure but tighter bounds.
Document-level grouping (strongest): Treats each document as one unit. Coverage 6.0%, risk 6.8%. This matches how documents actually arrive but is "nearly vacuous" — too few documents to certify much.
When Conditioning on Structure Wins
A key finding: pre-specified metadata (e.g., field provenance taxonomy) outperforms pooled thresholds on Sonnet + CORD (p < 1e-4), but this win does not replicate on the same documents under Claude Haiku or Qwen. Conversely, on higher-accuracy corpora, pooled thresholds work better. The insight: conditioning helps exactly where a pooled threshold cannot earn a statistical certificate — when uncertainty is high. Learned scores subsume this benefit where data quality improves.
Practical Validation
A three-annotator human audit on accepted fields verified 1.3% risk against the 10% budget (Fleiss' kappa=0.83). A frozen configuration on untested Claude Haiku 4.5 held both risk levels. The method is open-source with reproducible, regression-gated procedures.
Mine your own.
Lode is a workbench, not a feed. Paste a YouTube URL. The model proposes a transcript, a set of quote-grounded snippets, a synthesis essay, and the fan-out. You decide what stays.