How is this different from document QA benchmarks?
Document QA benchmarks usually measure accuracy on a fixed corpus. PDF Hell is a procedural stress test for named ingestion and rendering failure modes. Use both: a domain benchmark for representative quality and PDF Hell for targeted failure discovery.Why not use an LLM judge?
Each generator chooses the expected value before drawing the PDF. The scorer checks that procedural answer directly, so the same class of model being tested does not decide correctness. Optional explanations may use an evaluator, but they are outside the primary scoring path.What does fell_for_trap mean?
A case can record known lure answers, such as the invisible OCR amount. Matching
one marks the failure as trap-specific. It remains a diagnostic field; the
procedural expected answer determines correctness.
Can public generators be memorized?
The generators and registry are public for auditability. A fixed public suite can be contaminated, so use fresh deterministic seeds for private follow-up testing and disclose them when publishing. Memorizing one seed does not prove the family is solved.Which suite should I run?
smoke: 3 cases, wiring only.mini: 30 cases from the original three families.mini-v4-sample: 170 cases across all 17 current families; practical routine comparison.mini-v4: 510 cases across all 17 families; greater per-family precision.
mini-v2 and mini-v3 registries remain available for
historical reproduction. Always report the suite hash, not only its nickname.
What is pixels-only mode?
Normal PDF APIs may combine text extraction and rendering in an opaque provider pipeline.--pixels rasterizes locally and sends only images, isolating visual
reading more closely. It requires pdfhell[pixels]; the default is 150 DPI.
PDF and pixels-only scores are separate experiments and should not be pooled.
Are all historical leaderboard claims still valid?
No. An earlier Claude Opus result was produced by deprecated-temperature API errors that were silently counted as wrong answers. The repository’s correction notice supersedes the affected 0.4/0.5 README and confirmation claims. Current runs distinguish API errors from wrong answers. Two Unicode families also changed in 0.6.1:zero_width_space_split and unicode_confusable_total previously could render
visible replacement boxes. Only results generated with the redesigned families
support current invisible/confusable-character claims.
How do I publish a defensible result?
Include the package version, suite version and hash, model identifier, date, modality, DPI/rasterizer build when applicable, case count, pass rate with uncertainty, API error/refusal rates, and the raw JSON. Keep the audit pack when the result may be used in procurement or compliance decisions.Can I add a trap?
Yes. A useful generator is deterministic, code-grounded, answerable, free of forbidden-answer leakage, and meaningfully discriminative on fresh seeds. The optionalpdfhell[research] workflow proposes candidates, runs validation
gates, reports results, and creates a promotion plan; a human still decides what
enters a released suite.