CI integration - Multivon Docs

PDF Hell can fail a job on pass rate, emit JUnit XML, save the full JSON run, and bundle the PDFs and manifest into a hash-chained audit ZIP.

GitHub Actions

name: PDF Hell
on: [pull_request]

jobs:
  pdfhell:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - run: |
          uvx pdfhell run \
            --model anthropic:claude-haiku-4-5 \
            --suite mini-v4-sample \
            --out pdfhell-results.json \
            --junit pdfhell-results.xml \
            --audit-pack pdfhell-audit.zip \
            --fail-threshold 0.80
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: pdfhell-results
          path: |
            pdfhell-results.json
            pdfhell-results.xml
            pdfhell-audit.zip

Use if: always() so failure evidence is uploaded even when the threshold gate returns nonzero.

GitLab CI

pdfhell:
  image: python:3.12
  script:
    - pip install pdfhell
    - pdfhell run --model openai:gpt-4o --suite mini-v4-sample
        --out pdfhell-results.json
        --junit pdfhell-results.xml
        --fail-threshold 0.80
  artifacts:
    when: always
    paths:
      - pdfhell-results.json
    reports:
      junit: pdfhell-results.xml

Choose the gate deliberately

Start with smoke to verify credentials and transport.
Use mini-v4-sample (170 cases) for routine regression checks.
Use mini-v4 (510 cases) when the additional per-family precision justifies the cost and runtime.
Set --fail-threshold from your own established baseline and product risk, not from another model’s leaderboard score.
Pin the pdfhell version in CI and retain the run’s suite hash.

Compare like with like

A valid before/after comparison keeps all of these fixed:

suite hash;
provider and model identifier;
PDF versus pixels-only modality;
DPI and rasterizer build for pixels-only runs;
generation version and case files.

Do not silently replace failed API calls with wrong answers. Current reports separate API errors, refusals, and incorrect answers; investigate transport errors before treating a red gate as a model regression.

Evidence packs

--audit-pack path.zip packages the generated documents, answer keys, result data, and integrity manifest. Keep the JSON and audit ZIP together for procurement, compliance, or incident review. The JUnit file is optimized for CI rendering and is not a substitute for the full result artifact.

Pixels-only CI

pip install 'pdfhell[pixels]'
pdfhell run \
  --model google:gemini-2.5-flash \
  --suite mini-v4-sample \
  --pixels --dpi 150 \
  --out pixels-results.json \
  --fail-threshold 0.75

Name pixels artifacts explicitly and never compare their gate directly with a PDF-input baseline. They answer different questions about the model pipeline.

​GitHub Actions

​GitLab CI

​Choose the gate deliberately

​Compare like with like

​Evidence packs

​Pixels-only CI