Skip to main content
Use multivon-eval when you want a Python suite that can run locally, produce inspectable reports, and block a release when required evidence is missing. The goal is to determine whether a system change improves a specific task without unacceptable regressions. A large evaluator catalog alone cannot answer that question.

​
What is available today

  • Deterministic checks, LLM judges, and agent/conversation evaluators in one suite.
  • Separate quality failures, infrastructure errors, and skipped measurements.
  • Repeated runs, confidence intervals, and comparisons against saved reports.
  • Local HTML/JSON/JUnit artifacts, suite snapshots, and prompt-change tracking.
  • Bootstrap proposals to help start a suite, with thresholds and generated cases that still need review against task-specific examples.
Start with task success, then use the offline quickstart. For behavior changes in the current release, read the 0.19.0 migration guide.

​
Current workflow measurement: TAT-QA

On September 17, 2026, a paired Claude Sonnet 5 study covered all 1,663 questions in TAT-QA’s released test-gold file. The answer-only treatment reached 74.74% exact match and 82.76% F1. Requiring the same response to include table or paragraph evidence locations reduced those scores to 72.22% and 79.96%. The paired context-bootstrap intervals exclude zero in the negative direction. The joint response also increased estimated cost 41.70% and median latency 35.24%. Valid location syntax alone was insufficient: only 55.20% of questions met the strict rule of an exact answer, valid references and full agreement with released annotation locations. This result rejects the tested joint-generation prompt. It motivates a separate evidence-binding stage that cannot change the preserved answer, but that design still needs a new experiment. Read the study and practical implications or inspect the numeric artifacts and protocol. Public-label contamination, a mutable model alias and missing customer review prevent a SoTA or industrial-validity claim.

​
Current judge measurement: RAGChecker

On September 17, 2026, the existing AnswerAccuracy evaluator scored all 560 responses in RAGChecker’s 280 comparison cases using Haiku 4.5. Its Pearson correlation with overall human preference was 0.499, with a 95% case-bootstrap interval of 0.413–0.572. Two annotations per case were resampled together. A same-model direct rating scored 513/560 responses under its frozen integer-only parser. With the upstream missing-difference imputation, its correlation was 0.452. The paired difference interval crossed zero. A post-hoc leading-integer parser recovered 44 replies and raised the direct score to 0.533, reversing the point-estimate ordering. Neither analysis establishes QAG superiority. QAG used four calls per response; direct rating used one. The protocol, predictions and limitations are public. This is a maintainer-run study of one configuration, not a SoTA result or validation on regulated customer data. The sensitivity check is explicitly post-hoc and does not replace the frozen primary result.

​
Historical evidence: hallucination detection

These are maintainer-run measurements from earlier versions, not an independent ranking or a benchmark of 0.17.0. The new verdict parser and removal of refusal shortcuts can change results; the live benchmarks have not been rerun for this release. The QA slice contains 100 outputs from 50 source examples. HaluEval’s task subsets pair reference outputs with generated hallucinations; they are not all independently human-annotated examples. See the upstream dataset construction. Stored results. Multivon’s threshold was selected using this same QA slice. Different judge models also confound the DeepEval comparison. These measurements describe these configurations; they do not establish general QAG or framework superiority.

​
Cross-task result: HaluEval summarization

The QA-selected Hallucination threshold of 0.55 was applied to 60 outputs from 30 summarization source examples without retuning on that slice: Stored result. This is evidence of transfer to one other task slice. It does not establish performance on customer data or rule out exposure of a judge model to public benchmark data during training. The historical F1 interval [0.70–0.92] uses a case-level bootstrap reconstructed from aggregate counts. It does not account for shared source documents, threshold selection, judge variation, or parser changes. A new comparison should retain paired predictions and resample at the source-example level.

​
Costs and repeatability

A stored run of 50 cases with four LLM evaluators reported $0.00127 per case and 17.1 judge calls per case. This is a historical configuration-specific estimate, not a current price or latency promise. Run evidence. A cache reduces repeated calls for identical requests; it does not measure independent judge reliability. In 0.17.0 the explicit judge reliability sample bypasses the response cache. Current native Anthropic/OpenAI judge paths forward the configured temperature; provider/model constraints still apply. See judge configuration.

​
What remains to be demonstrated

We do not currently establish state-of-the-art evaluator accuracy. A credible claim needs independent held-out tasks, reviewed labels, matched judge and budget comparisons, repeated measurements, and error/skip coverage alongside quality metrics. Agent task completion also needs evidence of the resulting state, not only a convincing transcript. Use the library with your existing tracing or observability system. Compare alternatives on your own tasks, operational needs, and cost budget. The benchmark guide contains scripts, historical results, and methodological limitations.