Multimodal Evaluators

VQAFaithfulness and DocumentGrounding use a vision judge to assess text against supplied images. They are experimental heuristics. Their scores do not establish task completion, accurate perception, valid page citations or a safe release on their own.

Experimental in 0.19.0: the vision-qag/v2 protocol below corrects measurement handling in PyPI 0.18.0. Missing images and an empty extracted claim set are unmeasured; malformed or incomplete judge replies become judge errors. The prior 0.18.0 implementation can give an empty claim set a perfect score. Upgrade to 0.19.0 for the corrected behavior. No new threshold calibration or judge-accuracy claim accompanies this fix.

Supply images

Both evaluators read caller-supplied case.metadata:

image_path: one local image path.
image_url: one HTTP(S) URL or base64 data URI.
images: an ordered list of image paths, URLs or data URIs, for example one rendered image per document page. This key takes precedence when present; an empty list means no media. Do not supply both single-image keys.

An image list must contain nonempty strings. Dictionaries, sets and unordered iterables are rejected. Native PDFs, audio and video are not accepted by these two evaluators. Use Inspect’s multimodal input support for native execution and logs; render PDF pages explicitly when using these image-only graders. Local paths are read with the caller’s filesystem access. HTTP(S) URLs may be fetched by the Anthropic/OpenAI provider. This Google adapter accepts local paths and data URIs only. These legacy path/URL fields do not hash or freeze media contents; retain the original bytes separately for reproducibility. The separate content-bound media bridge provides verified native Inspect inputs; these legacy graders do not automatically consume that profile. Do not populate them from untrusted model output without authorizing the access.

Choose a judge and threshold

The adapters support the anthropic, openai and google providers. Configure a model whose endpoint supports image input. Known text-only model names are rejected early; an unknown or private model is passed to its provider for validation. This is not a tested model-compatibility matrix. An explicit threshold is a task decision. Without one, the existing calibration fallback policy applies, usually warning and returning 0.7; that value is not evidence of accuracy on your documents. Use development cases and independently reviewed held-out sources to validate a threshold. See review calibration.

from multivon_eval import (
    AcceptancePolicy, CheckRequirement, EvalCase, EvalSuite,
    JudgeConfig, VQAFaithfulness,
)

case = EvalCase(
    input="Read the invoice total.",
    case_id="invoice-001",
    source_id="invoice-source-001",
    metadata={"image_path": "invoice-page-1.png"},
)
grader = VQAFaithfulness(
    threshold=1.0,  # Example: require every extracted claim to receive Yes.
    judge=JudgeConfig(provider="anthropic", model="claude-haiku-4-5-20251001"),
)
report = EvalSuite("invoice description").add_case(case).add_evaluator(grader).run(
    lambda prompt: "The invoice total is 120.00.",  # Replace with your application.
)
decision = AcceptancePolicy((CheckRequirement("vqa_faithfulness"),)).evaluate(report)
print(decision.decision)

This example checks a supplied answer; it does not run an invoice extraction application or validate the amount with an independent oracle. Requiring the check in an acceptance policy prevents skipped/error measurements from silently satisfying the gate. Add independent task and state checks for real releases.

VQAFaithfulness

VQAFaithfulness(threshold: float | None = None, judge: JudgeConfig | None = None)

The judge extracts up to three distinct factual claims, then answers Yes or No for each. The score is the fraction confirmed. A valid extraction is a complete JSON array of nonempty strings; one complete JSON Markdown fence is permitted. The evaluator does not search a refusal or arbitrary prose for a plausible array. A verdict must be a single Yes/No token, optionally followed by a period or exclamation mark. Ambiguous wording is a judge error. Successful results retain the extracted claims, raw responses, threshold and protocol version in EvalResult.metadata. Reasons retain the complete claims. No extracted claims means skipped, including answers that abstain. Judge exceptions and invalid responses raise JudgeUnavailable; EvalSuite records them as judge_error, outside the measured quality denominator. Claim selection can omit important errors, and the same judge generates and checks the claims. This is not independent verification or an answer-completeness metric. Prompt wording that treats source text as data does not prove resistance to prompt injection.

DocumentGrounding

DocumentGrounding(threshold: float | None = None, judge: JudgeConfig | None = None)

Supply page images in metadata={"images": ["page-1.png", "page-2.png"]}. The judge answers three fixed questions about factual support, invented entities and handling of an exception/caveat. All three unique Q1, Q2, Q3 answers must be present and unambiguous before any score is computed. Missing, repeated or contradictory lines are judge errors, even with a threshold of zero. Valid results retain all three answers and the raw judge response. The score is the fraction of Yes answers. A document without an applicable exception exposes a limitation of the fixed rubric; Q3 has no task-specific applicability oracle. The grader neither checks citations nor identifies the supporting page or region. It has no demonstrated diagnostic advantage for legal, medical or financial work.

Provider and evidence limits

These legacy adapters make direct SDK calls. They do not yet provide complete request/usage capture, reliable billing reconciliation, media content identity, or the full JudgeConfig cache/retry/timeout behavior of other judge paths. The Anthropic adapter uses the SDK’s documented sampling-parameter migration and honors the configured request timeout. It sends the resolved temperature through extra_body; models that reject that setting return a judge error. Successful parsing is not a measurement of judge correctness. Retain native execution logs and validate providers separately before relying on a workflow. pdfhell supplies controlled PDF fixtures and code-derived oracles for supported scenarios. Keep those task assertions separate from vision judge scores, and distinguish text extraction from pixel perception. The document-to-ledger study records that distinction and the limits of its task mapping. MCP tools expose these graders too; their behavior follows the installed core version. See the tool reference.

Getting Started

Evaluators

Claude Code Skills

Guides

Reference

Compliance

Supply images

Choose a judge and threshold

VQAFaithfulness

DocumentGrounding

Provider and evidence limits

​Supply images

​Choose a judge and threshold

​VQAFaithfulness

​DocumentGrounding

​Provider and evidence limits

Supply images

Choose a judge and threshold

VQAFaithfulness

DocumentGrounding

Provider and evidence limits