Experiment records every suite run locally and lets you compare results across model versions, prompt changes, or time. No cloud, no account — stored as JSONL in ~/.multivon/experiments/.
Basic usage
Interpret a comparison
exp.compare() prints before/after metrics, coverage counts, confidence
intervals, and approximate significance tests. These records are summaries:
they do not preserve per-case pairing. A higher pass rate with more errors
may reflect missing evidence instead of an improvement.
For a paired comparison, keep the full reports:
View run history
Tags
Tags are free-form key-value pairs — use them to track anything meaningful:CLI
Storage
Runs are stored at~/.multivon/experiments/<name>.jsonl. Each line is a JSON object with the run summary — not the full case-by-case output. Use report.save_json() separately if you want the full results.
CI/CD integration
Save the full report before a gate can raise, then record it even on gate failure:always() CI step as shown in CI/CD.
