VQAFaithfulness and DocumentGrounding use a vision judge to assess text
against supplied images. They are experimental heuristics. Their scores do not
establish task completion, accurate perception, valid page citations or a safe
release on their own.
Experimental in 0.19.0: the
vision-qag/v2 protocol below corrects
measurement handling in PyPI 0.18.0. Missing images and an empty extracted claim
set are unmeasured; malformed or incomplete judge replies become judge errors.
The prior 0.18.0 implementation can give an empty claim set a perfect score.
Upgrade to 0.19.0 for the corrected behavior.
No new threshold calibration or judge-accuracy claim accompanies this fix.Supply images
Bind the bytes you want graded, and pass a resolver that produces them. The descriptor becomes part of case identity, and each artifact’s bytes are re-hashed and re-probed before the judge sees them, so a run records which bytes were graded:media_bound so a report can tell the two paths apart. Bound
audio and video are rejected by name: render or transcribe them to an image or
PDF page first.
Unbound references (deprecated)
Without bound media, both evaluators still read caller-suppliedcase.metadata
and emit a DeprecationWarning, because these fields bind no bytes:
image_path: one local image path.image_url: one HTTP(S) URL or base64 data URI.images: an ordered list of image paths, URLs or data URIs, for example one rendered image per document page. This key takes precedence when present; an empty list means no media. Do not supply both single-image keys.
Choose a judge and threshold
The adapters support theanthropic, openai, google and ollama providers,
the last keeping evaluation data on your own machine. Configure a model whose
endpoint supports image input. Known text-only model names are rejected early;
an unknown or private model is passed to its provider for validation. This is
not a tested model-compatibility matrix.
An explicit threshold is a task decision. Without one, the existing calibration
fallback policy applies, usually warning and returning 0.7; that value is not
evidence of accuracy on your documents. Use development cases and independently
reviewed held-out sources to validate a threshold. See
review calibration.
VQAFaithfulness
EvalResult.metadata. Reasons retain the complete claims.
No extracted claims means skipped, including answers that abstain. Judge
exceptions and invalid responses raise JudgeUnavailable; EvalSuite records
them as judge_error, outside the measured quality denominator.
Claim selection can omit important errors, and the same judge generates and
checks the claims. This is not independent verification or an answer-completeness
metric. Prompt wording that treats source text as data does not prove resistance
to prompt injection.
DocumentGrounding
metadata={"images": ["page-1.png", "page-2.png"]}.
The judge answers three fixed questions about factual support, invented entities
and handling of an exception/caveat. All three unique Q1, Q2, Q3 answers
must be present and unambiguous before any score is computed. Missing, repeated
or contradictory lines are judge errors, even with a threshold of zero.
Valid results retain all three answers and the raw judge response. The score is
the fraction of Yes answers. A document without an applicable exception exposes
a limitation of the fixed rubric; Q3 has no task-specific applicability oracle.
The grader neither checks citations nor identifies the supporting page or region.
It has no demonstrated diagnostic advantage for legal, medical or financial work.
Provider and evidence limits
These legacy adapters make direct SDK calls. They do not yet provide complete request/usage capture, reliable billing reconciliation, media content identity, or the fullJudgeConfig cache/retry/timeout behavior of other judge paths.
The Anthropic adapter uses the SDK’s
documented sampling-parameter migration
and honors the configured request timeout. It sends the resolved temperature
through extra_body; models that reject that setting return a judge error.
Successful parsing is not a measurement of judge correctness. Retain native
execution logs and validate providers separately before relying on a workflow.
pdfhell supplies controlled PDF fixtures and code-derived
oracles for supported scenarios. Keep those task assertions separate from vision
judge scores, and distinguish text extraction from pixel perception. The
document-to-ledger study records that distinction
and the limits of its task mapping.
MCP tools expose these graders too; their behavior follows the installed core
version. See the tool reference.
