Skip to main content
PDF Hell stress-tests AI document readers with PDFs generated from code. The generator chooses the answer before drawing each document, so scoring uses procedural ground truth rather than an LLM judge.

​
Install and authenticate

You can also run without a permanent install by replacing pdfhell with uvx pdfhell. Use the environment variable for your chosen provider: ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_API_KEY.

​
Run a suite

The runner materializes missing cases, sends each PDF to the selected model, and writes a JSON report under runs/. Use --out, --junit, --audit-pack, and --fail-threshold to make those artifacts explicit.

​
Inspect one trap

The same family and seed reproduce the same PDF and answer key.

​
Isolate visual reading

Provider PDF APIs may use the embedded text layer, rendered pixels, or both. Pixels-only mode rasterizes locally and sends PNG pages instead:
The default is 150 DPI; override it with --dpi. Reports record the modality, DPI, and pdfium build. Do not compare PDF and pixels-only scores as though they were the same experiment.
Use PDF Hell 0.6.1 or newer for zero_width_space_split and unicode_confusable_total. Earlier generators could render visible replacement glyphs, so old results for those two families are not comparable with current runs.

​
Discover new traps

The optional research package provides the budgeted autoresearch loop used to propose and validate new families:
Start with mini-v4-sample for routine comparisons and use mini-v4 when you need tighter per-family estimates. Always retain the suite hash from the report.