Skip to main content
The eval-framework category has converged on a small set of primitives, so feature checkmarks won’t separate the options. The question worth asking is: if I run the same task with my judge, do the numbers come out better? Below is what we can show from the public OSS repo. Every number links to a JSON file you can rerun.

​
Hallucination detection — HaluEval QA, N=100, human labels

All runs use claude-haiku-4-5-20251001 as the judge (source). What this means. Binary yes/no questions (QAG) are a more reliable scoring signal than numeric rubrics. The simple 1-10 judge ships ~3× more false positives at the same precision — every false positive in a CI gate is wasted developer time.

​
Cross-distribution held-out F1 — HaluEval-Sum, N=60

The in-distribution F1 above is the headline; the held-out number is the answer to “did you tune thresholds on the data you’re testing against?” The Hallucination evaluator’s threshold is calibrated on HaluEval-QA (threshold 0.55 for claude-haiku-4-5). We then test it against a different task family — HaluEval-Sum (summarization hallucination) — without changing anything else. Source — benchmarks/results/hallucination_held_out.json. A calibrated evaluator that holds its precision (0.957) when moved to a different task distribution is the actual product claim. Anyone can score well on the validation set they tuned against.

​
Three-framework agreement — κ=0.03

We ran multivon-eval, DeepEval, and RAGAS on the same 100 HaluEval-QA cases. Pairwise Cohen’s κ across all three pairings: κ=0.03, essentially independent. Despite the marketing copy, the frameworks are not measuring the same thing on the same task. Pick the one whose calibration trail and held-out generalization you can audit.

​
The 0.9.4→0.9.7 self-correction sequence

A framework that publishes numbers has to publish corrections. The four-release sequence below is the discipline we hold ourselves to:
  • 0.9.5 caught that 0.9.4’s “held-out HaluEval-Sum F1 0.783” was actually in-distribution — Faithfulness’s threshold is calibrated on HaluEval-Sum, so testing it back against HaluEval-Sum is leakage. Relabeled with a correction note same-day. CHANGELOG 0.9.5.
  • 0.9.6 noticed the eval_suite.py emitted by bootstrap called suite.run(cases=...) (non-existent kwarg) and report.print_summary() (non-existent method) — both fixed, regression test added.
  • 0.9.7 caught that 0.9.5’s held-out test was reporting threshold 0.7 in stderr but the calibrated value for Haiku on Hallucination is 0.55. Same dataset, same model — different F1 (0.852 vs 0.830). The threshold-vs-default gotcha is now an explicit reproducibility note in benchmarks/README.md.
  • 0.9.8 propagated the corrected numbers and the κ=0.03 finding to the README.
Same-day correction with a public trail is the standard we try to hold.

​
Multi-judge agreement — HaluEval QA, N=50, temp=0

Different judges disagree more than you’d expect. The calibrated-thresholds layer matters precisely because the underlying judge is non-uniform. Source. Pairwise Cohen’s κ: 0.60–0.80 — substantial agreement on most pairs. gemini-2.5-flash leads every metric in this run; claude-haiku-4-5 and gpt-4o-mini are close seconds with cheaper tokens. Pick by your cost / latency / sovereignty constraints — calibrated thresholds ship for each. claude-sonnet-4-6 is a useful diversity judge in multi-judge runs, not a default.

​
Cost — 50 cases × 4 LLM-judge evaluators

workers=1 (sequential), real Anthropic API. Source. QAG generates multiple yes/no questions per criterion then verifies each — so 4 evaluators ≈ 17 LLM calls. Trade-off is fully auditable scoring (every question / answer is in the report) for a few cents per case.

​
Cache speedup on re-runs

Same suite, sequential, with set_cache(JudgeCache(...)): Speedup: 2,271× — read that as paid API calls vs local cache hits (4 → 0), expected by construction, not a model-quality claim. CI re-runs (same git SHA + same dataset) converge to zero LLM calls. set_cache() auto-enables caching for every JudgeConfig — no need to thread cache=True through every evaluator.

​
Where competitors lead

We’re not better at everything.
  • If you want the widest evaluator catalog, DeepEval has more pre-built metrics for niche tasks (e.g. summarization-specific G-Eval variants).
  • If you want a vendor-managed cloud UI: DeepEval (Confident AI) and Promptfoo Cloud both ship hosted dashboards. We’re SDK-first, and the HTML viewer is local-only.
  • For pure prompt-comparison testing — “which prompt template wins on these N cases” — Promptfoo is purpose-built for that single job.

​
What multivon-eval is built for

  • Trusting the score. QAG plus calibrated thresholds plus multi-run flakiness detection means a single number from pass_rate survives scrutiny.
  • CI/CD on every PR. multivon-eval init --ci github ships the workflow, with distinct exit codes for quality vs infra failures.
  • Regulated AI. Hash-chained NDJSON audit logs with Article-level EU AI Act / NIST AI RMF / HIPAA mappings. audit-package produces an auditor-attachable zip; download a real sample (5.5 KB).
  • Agents. Tool-call accuracy, trajectory efficiency, and step faithfulness, framework-agnostic via AgentTracer.
  • Multi-judge setups. Ships with anthropic, openai, google, and litellm providers, plus any OpenAI-compatible endpoint (Ollama, vLLM, LM Studio, Azure, Bedrock via LiteLLM). Threshold packs are calibrated per (judge × evaluator), so you can swap providers without re-tuning.

​
Reproduce everything

All datasets are public. Judge model versions are pinned. If a number on this page diverges from what you measure, open an issue — we’ll fix it.
Comparison numbers reflect each project’s public releases as of July 2026. All CIs are Wilson 95% on precision/recall and 1000-resample bootstrap 95% on F1 (seed 20260603).

Footnotes

  1. In-distribution number — HaluEval-QA, the same dataset whose distribution thresholds are tuned against. See the held-out cross-distribution result in the next section. ↩