Extend existing interfaces
No plugin discovery registry is required. Import and compose Python objects.
The
_call_with_case and _acall_with_case hooks are internal compatibility
hooks; third-party code should not assume they form a stable public protocol.
For full-case task inputs, prefer native Inspect samples and solvers.
Evaluators can execute concurrently across cases. Avoid mutable per-case state
on a shared grader. The default aevaluate delegates synchronous grading to a
worker thread; cancelling an await cannot forcibly stop that thread. If a grader
implements prepare, it must be idempotent, and direct evaluate calls must
still work: saved-output and direct-call paths do not promise a warmup call.
Use a finite score in [0, 1]. Missing required evidence should produce
_skipped(reason); an unavailable judge should raise JudgeUnavailable.
Quality failure, missing evidence and execution failure have different meanings.
See custom graders,
environment outcomes and
versioned evidence.
Migrate imported-output replay
CaseImporter.as_model_fn previously consumed outputs in call order. That
could pair the wrong output with a case under reordering, concurrency or repeat
runs. Development builds match case identity and warn that the method is
deprecated. Direct string calls reject ambiguous prompts; missing outputs raise
an error. An explicitly recorded empty string remains valid.
Use saved-output grading instead:
latencies_ms, latency is unknown and
latency graders skip. Replaying a callable measures replay overhead; repeating
one saved response cannot measure target stochasticity. Judge-based graders may
still make API calls even though the target is not rerun.
Report format contract
EvalReport.to_json() writes multivon.report/v2. EvalReport.from_dict()
reads v2, explicit v1 and schema-less legacy v1. Missing legacy identities,
trials, provider usage and target settings remain missing. An unknown report
version or status, malformed fields and nonfinite numbers raise ValueError.
An advertised case outcome that contradicts the retained result fields is also
rejected instead of silently changing meaning when loaded.
The packaged envelope schema uses
JSON Schema Draft 2020-12, through the
existing jsonschema dependency. It does not fetch schemas over the network.
Dependencies and tested configurations
Heavy integrations use standard Python extras. Install only the extras required by your application. The base package includes Anthropic/OpenAI SDKs and JSON Schema, but does not require dataset engines, simulators, media decoders or ML training packages.
These are tested combinations, not evidence for every version allowed by the
dependency ranges. The core CI matrix targets Python 3.10–3.14; optional tests
skip when their dependencies are absent. Extra-specific checks must be run
before releasing changes to those integrations. The historical
all extra
contains the older model/browser/agent integrations; select the extras above
explicitly for these development integrations.
Contributor fixtures live in tests/test_extension_contracts.py and
tests/fixtures/reports/. See the repository’s
contribution guide
for installed-wheel and release checks. A future 1.0 stability promise requires
more downstream adoption evidence; development availability is not that promise.