Skip to main content
Available in multivon-eval 0.18.0. Tested with Inspect 0.3.263 and Hugging Face Datasets 5.0.1 on Python 3.10 and 3.12. Use Inspect when you need its model providers, agent solvers, sandboxes, durable evaluation logs, or retry/resume workflow. Multivon supplies compatible graders and task-specific acceptance decisions. Inspect owns execution; its native log remains the authoritative artifact.
This command is for a development checkout. Keep these optional dependencies out of small deployments that only need Multivon’s native offline runner.

​
An offline integration example

The fixture performs no provider calls. For a real application, use an Inspect solver/agent and model, and define a registered @task factory so Inspect can reconstruct it for retries. Consult Inspect’s log and retry documentation for eval_retry, native request logging, and sample preservation. A local crash experiment kills a process after a SQLite ledger write, preserves completed samples, and distinguishes safe replay from duplicate writes. It uses three synthetic cases per handler and no model API calls; it is not a production recovery guarantee.

​
Evidence mapping

  • Case IDs, full definitions, and manifest digests become native sample metadata.
  • Text context is sent as a system message; conversation messages precede the current input. Multimodal content needs explicit native task configuration.
  • Graders receive the actual native conversation and a projection of assistant tool calls and tool responses. An authored static trace cannot stand in for the actual execution trace.
  • Skip values remain null in Inspect scores. Grader exceptions remain sample errors, rather than becoming failed quality judgments.
  • Imported epochs are grouped by case. Each trial records the native log location, sample identity, sample digest, and model usage.
  • Incomplete upstream runs carry report-level evidence issues and cannot pass a Multivon acceptance policy.
For a retry chain, call from_inspect_log(final_log, previous_logs=[earlier_log]) with earlier logs in chronological order. Preserved sample UUIDs are deduplicated; new executions remain distinct. trial_scope="final_attempt" selects the last attempt per epoch on imported Inspect evidence. The default all-attempts policy includes interrupted attempts and therefore can remain indeterminate after a successful retry. The bridge cannot discover omitted historical logs.

​
Retry compatibility (development)

Native retry preservation by sample ID is not evidence that a task or grader definition stayed unchanged. Use bind_inspect_task inside the registered task factory, every time it is reconstructed:
This local example does not retry a failed task. In an actual registered factory, load the expected prior native log from a caller-selected path when reconstructing for eval_retry. The complete registered fixture and driver demonstrate that wiring, native sample preservation, a rejected preflight and an unguarded negative control. Binding only the first in-memory Task cannot guard a later factory reconstruction. The frozen study and raw evidence record all three cases: native mixed scores pass 3/3 while fresh changed-rule grading passes 2/3; the guarded retry adds zero target calls. The function updates the Task and sample metadata in place. It reuses existing grader/dependency fingerprints, engine inventory and named-file hashes; it does not add a scheduler, checkpoint store or dataset format. Static datasets need explicit unique sample IDs and bridge metadata. Graders need unique names and as_inspect_scorer; custom graders also need declare_dependencies. Use files={"policy": policy_path, "task_source": source_path} for immutable policy/code/configuration inputs that must be rehashed. Use dependencies for external revision identifiers. Keep runtime state such as ledger rows separate from immutable code/configuration inputs. The caller’s version and configuration must cover hidden solver arguments, environment/sandbox image revisions, model clients, services and closure state. A matching declaration is an assertion about that state, not automatic discovery or proof of immutability. The preflight compares the recorded contract before Inspect executes solvers. Task construction and grader preparation occur before binding and may already have side effects. Native eval/eval_retry overrides are not all available to the factory; keep them in the declared configuration and supply the complete log chain when importing. Native task/plan/execution changes across supplied logs also make acceptance indeterminate. Logging-only settings may differ. Each scorer records configuration before and after grading. Import checks sample bindings, native input/reference identity, observed grader drift and mixed task definitions. Compatibility issues remain in trial evidence through regrading. An incompatible native retry can therefore be diagnosed even if the caller omitted the preflight and Inspect already reused completed samples. Legacy retry chains without declarations are still readable, but their compatibility is unknown and cannot pass acceptance. Start a new declared evaluation; do not retroactively invent contracts for historical logs. Digests detect accidental changes, not forged evidence. Snapshots cannot detect transient changes restored between observations or establish arbitrary in-memory/sandbox checkpoint safety. Use Inspect View for the full native conversation and events. The adapter does not implement another generic log UI.

​
Execution limits and completion evidence (development)

Use Inspect’s sample limits and concurrency controls directly. For example, the offline task above can run with explicit controls:
These are example values, not production recommendations. max_samples bounds simultaneous samples; max_connections bounds upstream model connections. Inspect still scores the partial output after a sample limit stops execution. A native log marked success therefore does not imply every task completed. Development imports retain the native stop reason, invalidation, selected resource/generation settings and measured durations under each trial’s upstream.execution. Unknown completion after a limit produces an infrastructure error in the bridge and an indeterminate acceptance decision, while retaining all grader scores. Native invalidations and errors also remain blocking evidence. Regrading saved text cannot clear these execution constraints. If the task was explicitly defined to score output at a token boundary, declare that policy when importing:
This declaration permits that limit type; it does not establish that the task succeeded. Require independent outcome checks and apply the same declared policy to the entire retry chain. Unknown limit types are rejected. A time, operator or other stop remains blocking unless its type is also explicitly accepted; errors and invalidations cannot be overridden with accepted_limits. Increasing the general error budget does not authorize a new stop boundary or restore invalidated measurements. The offline control experiment uses Inspect’s actual runtime and mock provider plus a real SQLite ledger. It contrasts an acknowledgment with an independent persisted-state query, and includes a completed positive control. Token and cost limits check observed usage; an in-flight generation can exceed the threshold. The tested Inspect 0.3.263 turn_limit=1 path made two mock generations before stopping. These controls must not be presented as strict request reservations or hard monetary ceilings. The validation study and raw evidence retain all seven ledger scenarios, concurrency and cancellation results. Cancellation stops cooperative async work and retains native cancellation/error logs; it does not undo completed remote calls or SQLite commits. Configure provider/tool timeouts and a safe replay policy for side effects. Worker threads and code that blocks or suppresses cancellation require process/sandbox controls.

​
Native offline runner controls (development)

EvalSuite.run accepts positive integer workers and runs; workers=None selects the documented default. run_async also requires positive integer concurrency and, when set, evaluator_concurrency. The latter bounds evaluators across the whole run; it is not multiplied by the number of cases. Invalid controls and nonfinite/out-of-range quality gates fail before preparation or calls. On cancellation or an escaping child error, the async runner cancels and awaits its owned async tasks without cancelling unrelated application tasks. Default synchronous graders execute in threads: their awaits can be cancelled, but Python cannot terminate the underlying thread. Use cooperative async graders/provider timeouts or upstream sandbox isolation where stopping work is a requirement. The native runner does not add durable resumption or a second deadline scheduler; use Inspect for those execution facilities. Post-run assert_budget is a gate, not an execution spend cap.

​
Current boundaries

The importer requires samples and scorer metadata produced by this bridge. Arbitrary native scorers need a deliberate score mapping; their numbers are not silently treated as compatible Multivon verdicts. Native multimodal content and event timing remain in Inspect’s log; the text/tool projection is not lossless. Multivon LLM graders still use their configured judge providers. Those calls are not automatically covered by Inspect model-call logging or its model cost limits. Use deterministic graders until you explicitly account for judge requests and budgets. Imported sample errors are marked as infrastructure errors without claiming to distinguish model errors from grader errors. No average Inspect metric is registered by the scorer bridge: aggregating nullable scores without required-check coverage would hide missing evidence. Apply the acceptance policy after importing a log.