The problem
The question every team building with AI eventually hits: did this change make the model better or worse? Numeric scores drift, judges hallucinate, and regressions reach users before anyone notices. multivon-eval runs task-specific checks, records errors separately from quality failures, and compares results across system changes. Start by defining task success, then select graders and review their agreement with human judgments. Reports and audit artifacts support that decision; they do not certify safety or compliance.my_model_fn plus a provider
key. For a fully runnable first check without a key, see the quickstart.
The default judge is claude-haiku-4-5; configure another through JudgeConfig.
Historical benchmark results and their limits are in Why multivon-eval.
Want a runnable eval suite from a one-paragraph product description? multivon-eval bootstrap. Want it wired into Claude Code as auto-invoking skills? multivon-eval install-skills.
New in 0.17.0: skipped checks no longer count as passes; active quality
gates reject errors and incomplete coverage by default; judge reliability
checks bypass the response cache. Read the migration guide
before comparing new results with old reports.
What it does
Deterministic
String matching, regex, JSON schema, BLEU, ROUGE, latency. Instant, free, no LLM.
LLM-as-judge
QAG scoring through explicit yes/no questions. Faithfulness, hallucination, relevance, and more.
Agent trace
Tool call accuracy, plan quality, step faithfulness, task completion. Framework-agnostic.
Conversation
Knowledge retention, relevance, consistency, and completeness across multi-turn sessions.
Compliance
Local PII detection across GDPR, HIPAA, DPDP, and more, plus Pydantic / JSON Schema validation. Zero API calls.
Consistency
Self-consistency across repeated runs — catch answers that drift between identical prompts.
Multimodal
VQA faithfulness and document grounding for image and document inputs.
Key features
QAG scoring. Instead of asking a judge “rate this 1-10”, we generate binary yes/no questions about the output and score by fraction answered correctly. Binary questions reduce scale ambiguity. Inspect the generated questions and judge reasons; grading errors and omitted claims are still possible. Plain-English checks.suite.add_check("Response should mention the return policy") is all you need to write your first eval. The SDK generates yes/no questions from your criterion automatically — no evaluator class to pick, no QAG questions to craft.
No cold start, either: point generate_from_file() at your docs and get eval cases in seconds, no labeled dataset required.
Reliability and flakiness detection. Run each case N times with suite.run(runs=5) and flaky cases (inconsistent pass/fail) are flagged automatically. Comparison tests help assess sampling uncertainty under their assumptions. They do not detect every source of grader bias or dataset error.
For tracking over time, every run can be recorded to ~/.multivon/experiments/; compare two runs side-by-side and get a pass rate delta with p-values.
Framework integrations. Capture agent traces from LangChain, LangSmith, or any custom agent. Import existing LangSmith runs as eval cases without re-running the agent.
Shareable HTML reports. report.save_html("report.html") produces a self-contained HTML file with per-case breakdowns, evaluator scores, and flakiness indicators. No server required.
CI/CD first. An active quality gate exits 1 for a completed quality failure and 2 for indeterminate evidence. By default, errors and skipped coverage block the gate.
Prompt-drift staleness and provenance. multivon-eval staleness diffs a committed baseline of every prompt call site in your repo against a live scan and tells you which prompts changed since your cases were authored: CHANGED, REMOVED, ADDED, or UNKNOWN, never overclaiming what static analysis can know. An opt-in runtime recorder (pytest --record-prompts) captures the prompts your code actually rendered, with verdicts phrased as “matched k of N observed renderings”. See Prompt-drift staleness.
Persona simulation. multivon-eval simulate drives adaptive multi-turn conversations against your live system. A persona LLM with a profile, goal, and behavior traits generates each user turn in response to what your model actually said; transcripts are scored by the conversation evaluators plus a goal-completion judge, with a hard budget ceiling and every output labeled as synthetic. See Simulate.
Install
ANTHROPIC_API_KEY and/or OPENAI_API_KEY for LLM-judge evaluators.
Quickstart
Up and running in 5 minutes
Generate datasets
No labeled data? Generate cases from your docs