> ## Documentation Index
> Fetch the complete documentation index at: https://docs.multivon.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# CI/CD Integration

> Run evals as a quality gate in your deployment pipeline.

## The pattern

Pass `fail_threshold` to `suite.run()`. If the pass rate drops below it, the process exits with code 1 — blocking the deployment.

```python theme={null}
# eval.py
from multivon_eval import EvalSuite, EvalCase, load, Faithfulness, NotEmpty

suite = EvalSuite("Prod Eval")
suite.add_cases(load("tests/cases.jsonl"))
suite.add_evaluators(NotEmpty(), Faithfulness())

report = suite.run(
    my_model,
    fail_threshold=0.85,             # block deployment if < 85% pass
    save_json="eval_results.json",   # written BEFORE the gate fires
    save_html="eval_results.html",
)
```

<Warning>
  Save through the `save_json` / `save_html` / `save_junit_xml` **kwargs**, not
  by calling `report.save_json(...)` on the line after a gated `run()`. A
  failing gate raises `EvalGateFailure` from inside `run()`, so statements
  after it never execute — exactly the runs whose artifacts you need most.
  The kwargs write the report before the gate is evaluated.
</Warning>

## GitHub Actions

```yaml theme={null}
# .github/workflows/eval.yml
name: LLM Eval

on:
  push:
    branches: [main]
  pull_request:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install multivon-eval

      - name: Run evals
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: multivon-eval run eval.py --html eval_results.html --json eval_results.json

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: |
            eval_results.json
            eval_results.html
```

## Speed up CI with parallel workers

```python theme={null}
# Run 8 cases in parallel via threads
report = suite.run(model_fn, workers=8, fail_threshold=0.85)
```

## Async runner for async model functions

```python theme={null}
import asyncio
from multivon_eval import EvalSuite

async def my_async_model(input: str) -> str:
    ...

report = asyncio.run(
    suite.run_async(my_async_model, concurrency=10, fail_threshold=0.85)
)
```

## Save and view results

```python theme={null}
report.save_json("results.json")
report.save_csv("results.csv")
report.save_html("results.html")   # self-contained HTML report
```

Or get the HTML as a string (useful in notebooks or custom pipelines):

```python theme={null}
html = report.to_html()
```

View a saved report from the CLI:

```bash theme={null}
multivon-eval report results.json
```

Convert a saved JSON report to HTML:

```bash theme={null}
multivon-eval report results.json --html results.html
```

## Run a specific eval file

```bash theme={null}
multivon-eval run eval.py

# Save reports automatically
multivon-eval run eval.py --html results.html
multivon-eval run eval.py --html results.html --json results.json
```

`--html` and `--json` inject environment variables that the suite picks up automatically — no changes to your eval script needed.

## Tips

* Keep a golden dataset: a small set of cases (20-50) that cover your most critical behaviors, run on every PR.
* Use tags to split fast deterministic evals (every commit) from slower LLM-judge evals (nightly, or main only).
* Archive `results.json` per run as a build artifact so you can track score trends over time.