LLM Judge Evaluators

LLM judge evaluators use a secondary model to assess output quality. multivon-eval uses QAG scoring (Question-Answer Generation) — generating binary yes/no questions about the output instead of asking for a numeric 1-10 rating. Why QAG? Binary questions make the grading criteria inspectable and avoid numeric scale ambiguity. They can require several model calls per output. Accuracy and cost depend on the task, generated questions, and judge; QAG is not a general accuracy guarantee.

Configuration

JudgeConfig

The judge model is fully decoupled from your pipeline model. Configure it once globally, override per-evaluator, or fall back to environment variables.

from multivon_eval import configure, JudgeConfig, Faithfulness

# Set globally at startup — all evaluators use this unless overridden
configure(JudgeConfig(provider="openai", model="gpt-4o-mini"))

# Override for a specific evaluator
Faithfulness(judge=JudgeConfig(provider="anthropic", model="claude-opus-4-7"))

Resolution order (highest to lowest):

Per-evaluator judge= kwarg
configure() global
JUDGE_PROVIDER / JUDGE_MODEL environment variables
Built-in default: anthropic / claude-haiku-4-5

# Environment variable fallback
export ANTHROPIC_API_KEY=sk-ant-...
export JUDGE_PROVIDER=anthropic
export JUDGE_MODEL=claude-haiku-4-5

JudgeConfig field	Default	Description
`provider`	`"anthropic"`	`anthropic`, `openai`, `google`, `ollama`, or `litellm`
`model`	`"claude-haiku-4-5"`	Model name for the chosen provider
`base_url`	`""`	Custom endpoint for local/self-hosted servers (see below)
`temperature`	`None` = inherit, effective `0.0`	Configured sampling temperature; see transport limitation below
`max_tokens`	`None` = inherit, effective `1024`	Token budget for judge responses
`timeout`	`None` = inherit, effective `30`	Request timeout in seconds

Changed in 0.16.0: temperature, max_tokens, timeout, and reliability_sample now default to None, meaning “inherit from the global config” — they resolve to the same effective defaults as before. The old merge compared overrides against the default values, so an explicit JudgeConfig(temperature=0.0) was silently ignored whenever a nonzero global was configured. Explicit values — including 0.0 — now always win.

Fixed in 0.16.1 — reasoning-tier judges. The QAG evaluators used small per-call caps (100 tokens for a yes/no verdict, 512 for claim extraction). That is plenty for a plain-text judge, but a reasoning model spends part of its output budget thinking before it answers, so those caps cut it off mid-thought and it returned an empty verdict. We measured a 47% error rate running gpt-5.5 as the judge. The per-call ceiling is now floored at 2048 tokens when the judge model is reasoning-tier (gpt-5/o1/o3/o4 prefixes). Every other judge is unaffected — its cap passes through untouched. Evaluator-specific per-call caps take precedence over the global JudgeConfig.max_tokens; the reasoning-model floor applies to that per-call cap.

The model under test and the judge model can be different providers.

In 0.17.0 the native Anthropic and OpenAI text-judge request paths do not forward JudgeConfig.temperature; Ollama uses the OpenAI path and shares this limitation. Google and LiteLLM do forward it. The resolved configuration is therefore not a complete record of effective sampling behavior. Temperature zero also does not guarantee identical outputs. timeout is forwarded to supported clients.

Local and self-hosted models

Any OpenAI-compatible server works as a judge — Ollama, LM Studio, vLLM, llama.cpp, or a self-hosted endpoint:

# Ollama running locally
configure(JudgeConfig(
    provider="openai",
    model="llama3",
    base_url="http://localhost:11434/v1",
))

# LM Studio
configure(JudgeConfig(
    provider="openai",
    model="local-model",
    base_url="http://localhost:1234/v1",
))

# Self-hosted vLLM or any OpenAI-compatible endpoint
configure(JudgeConfig(
    provider="openai",
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    base_url="https://my-inference-server.internal/v1",
))

base_url is also read from the OPENAI_BASE_URL environment variable, so no code changes are needed to switch between cloud and local judges in CI.

Calibrated thresholds

Faithfulness, Hallucination, and Relevance look up historical threshold packs for the configured judge. These are starting points, not proof of calibration on your task. HaluEval task subsets include generated hallucinations; see the benchmark methodology. Validate thresholds against separate, human-reviewed development and test sets for your domain, especially after changing the judge or parser.

Judge	Hallucination	Faithfulness	Relevance
`claude-haiku-4-5-20251001`	0.55	0.90	0.30
`claude-sonnet-4-6`	0.30	0.90	0.30
`gpt-4o-mini`	0.30	0.90	0.30
Models without a calibration entry	0.70	0.70	0.70

This table is an excerpt; threshold_table() lists all shipped entries. Pass threshold= explicitly to override:

Faithfulness(threshold=0.8)   # use your own threshold, skip calibration

To inspect the full calibration table:

from multivon_eval import threshold_table
print(threshold_table())

UNKNOWN verdicts

New in 0.16.0. QAG scoring asks the judge binary yes/no questions — but judges hedge, and a hedge is not a verdict. The parser now has three outcomes per question:

In 0.17.0, a leading “yes”/“no” verdict or a complete explicit phrase such as “The answer is yes.” parses as a verdict. Mentions inside explanations (“I cannot say yes”) and “yes or no” are UNKNOWN. Historical threshold measurements predate this stricter parser and need revalidation.
A reply with no unambiguous verdict is UNKNOWN: excluded from the score denominator entirely and disclosed in the result reason, e.g. 1 of 3 question(s) UNKNOWN — excluded from score denominator. An UNKNOWN never counts for or against the model.
If every verdict for a case is unparseable, the evaluator raises JudgeUnavailable and the case gets JUDGE_ERROR status — excluded from pass_rate, counted in report.errors.

One honest sentence about the past: before 0.16.0 the parser fell back to “does ‘yes’ appear in the first 50 characters”, so a judge replying “I cannot say yes or no with certainty” was scored as YES. That yes-bias is gone; the previously mis-scored tail is now UNKNOWN.

Error budget: `max_error_rate`

pass_rate excludes errored cases by design — a judge outage is not a quality regression. Since 0.17.0, an active fail_threshold gate rejects any error by default. An explicit max_error_rate permits a chosen error budget:

from multivon_eval import EvalSuite, EvalCase, EvalGateFailure
from multivon_eval.evaluators.deterministic import Contains

suite = EvalSuite("checkout-bot")
suite.add_cases([EvalCase(input=f"question {i}") for i in range(10)])
suite.add_evaluators(Contains(["ok"]))

def mostly_down(prompt: str) -> str:
    if prompt != "question 0":
        raise TimeoutError("upstream 504")   # stands in for judge/model outages
    return "ok"

try:
    suite.run(mostly_down, fail_threshold=0.85, max_error_rate=0.10, verbose=False)
except EvalGateFailure as e:
    print(e)
# Eval gate INDETERMINATE: error rate 90.0% exceeds error budget 10.0% —
# 9 of 10 case(s) errored before quality could be scored (model_error=9).
# pass_rate 100.0% covers only the 1 evaluated case(s); fix the errors
# before trusting this gate.

Available on suite.run, run_async, and run_on_cases. With max_error_rate unset, an active fail_threshold gate uses a zero-error budget. Empty and skipped coverage also make the gate indeterminate (exit code 2). A completed quality failure exits 1. Runs with neither gate configured still return a report for inspection. report.error_rate exposes the number directly (denominator is total, not evaluated — this is exactly the metric pass_rate cannot see, so read the two together).

Faithfulness

Extracts factual claims and checks their support in the provided context. The extractor can omit claims, so even a perfect score does not prove full grounding. When to use: RAG pipelines, document Q&A, or any task where the model must answer strictly from a provided source.

from multivon_eval import EvalCase, Faithfulness

case = EvalCase(
    input="What does the report say about Q3 revenue?",
    context="Q3 revenue was $4.2M, up 18% YoY...",
)
Faithfulness()
Faithfulness(threshold=0.8)

Requires context on the EvalCase.

Parameter	Type	Default	Description
`threshold`	`float` or `None`	`None`	Historical judge-specific threshold, with the configured fallback; an explicit value overrides it
`max_claims` (development)	positive `int`	`10`	Maximum unique claims to verify; exceeding it makes the measurement unscored

Changed in 0.19.0: every unique extracted claim must receive a valid verdict. Empty extraction or exceeding max_claims returns a skipped measurement, not a pass. Invalid claim types raise an evaluator error; any unknown verdict raises JudgeUnavailable. Older versions could pass an empty list, score a ten-claim prefix, or omit unknown verdicts.

For development checkouts, Faithfulness(threshold=0.9, max_claims=30) allows up to 30 unique extracted claims. This can require one extraction call plus 30 verification calls before provider retries. Exact duplicates are collapsed after trimming whitespace. Results retain extracted/unique/verified counts, raw extraction and verdict replies, and the effective threshold. These fields expose the observed coverage, not extraction completeness. Use an AcceptancePolicy requiring faithfulness when missing measurements must block acceptance. A skipped faithfulness result can coexist with another passing check, so aggregate case pass rates alone do not enforce required-check coverage. Regrade saved outputs or rerun cases before using new results in comparisons; the old calibration packs have not been revalidated for this protocol.

Hallucination

Checks that the output doesn’t introduce claims not supported by context. When to use: Complement to Faithfulness — Faithfulness checks what’s present in the answer; Hallucination checks what shouldn’t be there.

from multivon_eval import EvalCase, Hallucination

case = EvalCase(
    input="Summarize the contract terms",
    context="Contract: 12-month term, $500/month, 30-day cancellation notice.",
)
Hallucination()
Hallucination(threshold=0.8)

Requires context.

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

Relevance

Checks that the output actually addresses the input question. When to use: Catch evasive, off-topic, or boilerplate responses that technically answer but don’t address what was asked.

from multivon_eval import EvalCase, Relevance

case = EvalCase(input="How do I reset my password?")
Relevance()
Relevance(threshold=0.7)

No context required.

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

Coherence

Checks that the output is clear, well-structured, and logically sound. When to use: Long-form generation (articles, reports, explanations) where logical flow and clarity matter.

from multivon_eval import EvalCase, Coherence

case = EvalCase(input="Explain how photosynthesis works")
Coherence()
Coherence(threshold=0.8)

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

Toxicity

Checks that the output is safe, non-harmful, and appropriate. When to use: User-facing products, content moderation pipelines, or any deployment where harmful output is unacceptable. Score 1.0 = not toxic; 0.0 = toxic.

from multivon_eval import EvalCase, Toxicity

case = EvalCase(input="Write a response to this angry customer message")
Toxicity()
Toxicity(threshold=0.9)

Parameter	Type	Default	Description
`threshold`	`float`	`0.9`	Minimum score to pass (higher default reflects zero-tolerance)

Bias

Checks that the output is free of demographic, political, or cultural bias. When to use: HR tools, news summarizers, recommendation systems, or any application where systematic favoritism is a risk. Score 1.0 = no bias detected; 0.0 = significant bias.

from multivon_eval import EvalCase, Bias

case = EvalCase(input="Describe the ideal job candidate for this role")
Bias()
Bias(threshold=0.8)

Parameter	Type	Default	Description
`threshold`	`float`	`0.8`	Minimum score to pass

Summarization

Checks that a summary captures the key points of the source faithfully, without adding or omitting critical information. When to use: Summarization pipelines — news, legal documents, meeting transcripts.

from multivon_eval import EvalCase, Summarization

case = EvalCase(
    input="Summarize this article",
    context="[Full source article text here...]",
)
Summarization()
Summarization(threshold=0.8)

Requires context (the source document).

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

AnswerAccuracy

Checks factual correctness of the output against expected_output. Uses judge comparison rather than string matching, so paraphrasing is handled correctly. When to use: Knowledge QA, fact retrieval, or any task with a known correct answer where the phrasing may vary.

from multivon_eval import EvalCase, AnswerAccuracy

case = EvalCase(
    input="What is the capital of France?",
    expected_output="Paris",
)
AnswerAccuracy()
AnswerAccuracy(threshold=0.8)

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

ContextPrecision

For RAG systems: checks that retrieved context chunks are actually relevant to the question. High precision = low noise in retrieval. When to use: Evaluating the retrieval stage of a RAG pipeline independently from generation.

from multivon_eval import EvalCase, ContextPrecision

case = EvalCase(
    input="What is our refund policy?",
    context=["Refund policy: 30 days...", "Shipping rates: ...", "Contact us at..."],
)
ContextPrecision()
ContextPrecision(threshold=0.8)

Accepts context as either a string or a list of strings (chunks). Evaluates up to 8 chunks.

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

ContextRecall

For RAG systems: checks that the retrieved context contains everything needed to derive the expected answer. When to use: Diagnosing retrieval gaps — cases where the model gave a wrong answer because the right chunk wasn’t retrieved.

from multivon_eval import EvalCase, ContextRecall

case = EvalCase(
    input="What is the cancellation fee?",
    context="Cancellation within 30 days: $50 fee applies.",
    expected_output="$50",
)
ContextRecall()
ContextRecall(threshold=0.8)

Requires both context and expected_output.

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

CustomRubric

Define your own yes/no criteria. Each criterion is a (question, expected_answer) tuple. Score = fraction of criteria where the judge’s answer matches expected_answer. When to use: Domain-specific quality checks that don’t map to the built-in evaluators — support tone, legal disclaimers, brand voice.

from multivon_eval import EvalCase, CustomRubric

case = EvalCase(input="Handle this support ticket: 'My order hasn't arrived'")
CustomRubric(
    name="support_quality",
    criteria=[
        ("Does the response acknowledge the customer's problem?", True),
        ("Does the response provide a concrete next step?", True),
        ("Does the response use apologetic or defensive language?", False),
        ("Is the response under 150 words?", True),
    ],
    threshold=0.75,
)

Parameter	Type	Default	Description
`criteria`	`list[tuple[str, bool]]`	required	List of `(question, expect_yes)` pairs
`name`	`str`	`"custom_rubric"`	Display name for this evaluator in reports
`threshold`	`float`	`0.7`	Minimum fraction of criteria to pass

GEval

Holistic numeric scoring for qualities that don’t decompose well into yes/no questions (creativity, tone, polish). The judge returns a 0.0–1.0 score directly with reasoning. When to use: Subjective qualities like writing style, creativity, or polish where binary questions don’t capture the nuance. Use sparingly — less auditable than QAG evaluators.

from multivon_eval import EvalCase, GEval

case = EvalCase(input="Write a product description for wireless headphones")
GEval(
    name="writing_quality",
    criteria="The response is engaging, concise, and professionally written.",
    threshold=0.7,
)

Parameter	Type	Default	Description
`criteria`	`str`	required	Free-text description of what to evaluate
`name`	`str`	`"g_eval"`	Display name for this evaluator in reports
`runs`	`int`	`2`	Number of judge runs to average — reduces position and framing bias
`judge`	`JudgeConfig`	`None`	Override the judge model for this evaluator
`threshold`	`float`	`0.7`	Minimum score to pass

GEval is the only evaluator that uses a numeric score directly from the judge rather than QAG aggregation.

CheckEvaluator

The fastest way to add a quality check. You write a plain-English criterion; CheckEvaluator auto-generates specific yes/no questions from it and scores with QAG. No need to pick an evaluator class or write questions manually.

from multivon_eval import EvalSuite, EvalCase

suite = EvalSuite("return policy eval")
suite.add_check("Response should mention the return policy")
suite.add_check("Tone should be professional and not defensive", threshold=0.8)
suite.add_cases([EvalCase(input="What is your return policy?")])
report = suite.run(my_model)

Questions are generated once at the start of suite.run() (eager warmup), so no case pays the generation cost and failures surface before the eval loop starts.

Escape hatch: pin questions for CI

Generated questions vary per run and per model. For reproducible CI runs, pin them explicitly:

suite.add_check(
    "Response should mention the return policy",
    questions=[
        "Does the response mention a return window?",
        "Does the response name the return policy by name?",
        "Does the response provide a link or next step?",
    ],
)

When questions= is set, no LLM call is made during prepare().

Inspect generated questions

import logging
logging.basicConfig(level=logging.INFO)  # prints generated questions to stdout

ev = suite._evaluators[0]
suite.run(my_model)
print(ev.resolved_questions)  # ['Does the response...', ...]

Discrete scores for N questions

With the default num_questions=3, the only possible scores are 0.0, 0.33, 0.67, and 1.0. The default threshold of 0.7 therefore requires 3/3 questions to pass. Lower the threshold or use num_questions=5 if you want more granularity.

`num_questions`	Possible scores	Threshold 0.7 requires
3	0.0, 0.33, 0.67, 1.0	3/3
5	0.0, 0.2, 0.4, 0.6, 0.8, 1.0	4/5
10	0.0, 0.1, …, 1.0	7/10

Fallback behavior

If question generation fails after two attempts, CheckEvaluator issues a warnings.warn and falls back to using the criterion itself as a single yes/no question. The EvalResult reason will include a [⚠ question generation failed — using fallback] tag. Check ev._used_fallback programmatically.

Parameter	Type	Default	Description
`criterion`	`str`	required	Plain-English quality criterion (max 300 chars)
`threshold`	`float`	`0.7`	Fraction of questions that must pass
`num_questions`	`int`	`3`	Number of yes/no questions to generate (clamped 1–10)
`questions`	`list[str]`	`None`	Pin specific questions — skips generation entirely
`name`	`str`	derived	Evaluator name in reports (auto-derived from criterion if omitted)
`judge`	`JudgeConfig`	`None`	Override the judge model for this evaluator

Getting Started

Evaluators

Claude Code Skills

Guides

Reference

Compliance

Configuration

JudgeConfig

Local and self-hosted models

Calibrated thresholds

UNKNOWN verdicts

Error budget: `max_error_rate`

Faithfulness

Hallucination

Relevance

Coherence

Toxicity

Bias

Summarization

AnswerAccuracy

ContextPrecision

ContextRecall

CustomRubric

GEval

CheckEvaluator

Escape hatch: pin questions for CI

Inspect generated questions

Discrete scores for N questions

Fallback behavior

​Configuration

​JudgeConfig

​Local and self-hosted models

​Calibrated thresholds

​UNKNOWN verdicts

​Error budget: max_error_rate

​Faithfulness

​Hallucination

​Relevance

​Coherence

​Toxicity

​Bias

​Summarization

​AnswerAccuracy

​ContextPrecision

​ContextRecall

​CustomRubric

​GEval

​CheckEvaluator

​Escape hatch: pin questions for CI

​Inspect generated questions

​Discrete scores for N questions

​Fallback behavior

Configuration

JudgeConfig

Local and self-hosted models

Calibrated thresholds

UNKNOWN verdicts

Error budget: `max_error_rate`

Faithfulness

Hallucination

Relevance

Coherence

Toxicity

Bias

Summarization

AnswerAccuracy

ContextPrecision

ContextRecall

CustomRubric

GEval

CheckEvaluator

Escape hatch: pin questions for CI

Inspect generated questions

Discrete scores for N questions

Fallback behavior