Skip to main content

​
The pattern

Pass fail_threshold to suite.run(). If the pass rate drops below it, the process exits with code 1 — blocking the deployment. 
# eval.py
from multivon_eval import EvalSuite, EvalCase, load, Faithfulness, NotEmpty

suite = EvalSuite("Prod Eval")
suite.add_cases(load("tests/cases.jsonl"))
suite.add_evaluators(NotEmpty(), Faithfulness())

report = suite.run(
    my_model,
    fail_threshold=0.85,             # block deployment if < 85% pass
    save_json="eval_results.json",   # written BEFORE the gate fires
    save_html="eval_results.html",
)
Save through the save_json / save_html / save_junit_xml kwargs, not by calling report.save_json(...) on the line after a gated run(). A failing gate raises EvalGateFailure from inside run(), so statements after it never execute — exactly the runs whose artifacts you need most. The kwargs write the report before the gate is evaluated.

​
GitHub Actions

# .github/workflows/eval.yml
name: LLM Eval

on:
  push:
    branches: [main]
  pull_request:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install multivon-eval

      - name: Run evals
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: multivon-eval run eval.py --html eval_results.html --json eval_results.json

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: |
            eval_results.json
            eval_results.html

​
Speed up CI with parallel workers

# Run 8 cases in parallel via threads
report = suite.run(model_fn, workers=8, fail_threshold=0.85)

​
Async runner for async model functions

import asyncio
from multivon_eval import EvalSuite

async def my_async_model(input: str) -> str:
    ...

report = asyncio.run(
    suite.run_async(my_async_model, concurrency=10, fail_threshold=0.85)
)

​
Save and view results

report.save_json("results.json")
report.save_csv("results.csv")
report.save_html("results.html")   # self-contained HTML report
Or get the HTML as a string (useful in notebooks or custom pipelines): 
html = report.to_html()
View a saved report from the CLI: 
multivon-eval report results.json
Convert a saved JSON report to HTML: 
multivon-eval report results.json --html results.html

​
Run a specific eval file

multivon-eval run eval.py

# Save reports automatically
multivon-eval run eval.py --html results.html
multivon-eval run eval.py --html results.html --json results.json
--html and --json inject environment variables that the suite picks up automatically — no changes to your eval script needed.

​
Tips

  • Keep a golden dataset: a small set of cases (20-50) that cover your most critical behaviors, run on every PR.
  • Use tags to split fast deterministic evals (every commit) from slower LLM-judge evals (nightly, or main only).
  • Archive results.json per run as a build artifact so you can track score trends over time.