> ## Documentation Index
> Fetch the complete documentation index at: https://docs.multivon.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Use Inspect for execution and logs

> Run Multivon graders inside Inspect and apply acceptance policies to native logs.

**Available in multivon-eval 0.18.0.** Tested with Inspect 0.3.263
and Hugging Face Datasets 5.0.1 on Python 3.10 and 3.12.

Use [Inspect](https://inspect.aisi.org.uk/) when you need its model providers,
agent solvers, sandboxes, durable evaluation logs, or retry/resume workflow.
Multivon supplies compatible graders and task-specific acceptance decisions.
Inspect owns execution; its native log remains the authoritative artifact.

```bash theme={null}
pip install -e '.[inspect,datasets]'
```

This command is for a development checkout. Keep these optional dependencies
out of small deployments that only need Multivon's native offline runner.

## An offline integration example

```python theme={null}
from inspect_ai import Task, eval as inspect_eval
from inspect_ai.model import ChatMessageAssistant, ModelOutput
from inspect_ai.solver import solver
from multivon_eval import (
    AcceptancePolicy, CaseManifest, CheckRequirement, EvalCase, ExactMatch,
)
from multivon_eval.integrations.inspect import (
    as_inspect_scorer, from_inspect_log, to_inspect_dataset,
)

@solver
def fixture_answer():
    async def solve(state, generate):
        state.output = ModelOutput.from_content("fixture", "hold")
        state.messages.append(ChatMessageAssistant(content="hold"))
        return state
    return solve

manifest = CaseManifest("authorization fixture", [
    EvalCase("Missing approval", "hold", case_id="approval-1", source_id="doc-1"),
])
task = Task(dataset=to_inspect_dataset(manifest), solver=fixture_answer(),
            scorer=as_inspect_scorer(ExactMatch()), epochs=2)
logs = inspect_eval(task, model="mockllm/model", log_dir="inspect-logs",
                    display="none", log_model_api=True)
report = from_inspect_log(logs[0])
decision = AcceptancePolicy((CheckRequirement("exact_match"),)).evaluate(report)
assert decision.decision == "accept"
assert len(report.case_results[0].trials) == 2
```

The fixture performs no provider calls. For a real application, use an Inspect
solver/agent and model, and define a registered `@task` factory so Inspect can
reconstruct it for retries. Consult [Inspect's log and retry documentation](https://inspect.aisi.org.uk/eval-logs.html)
for `eval_retry`, native request logging, and sample preservation. A
[local crash experiment](https://github.com/multivon-ai/multivon-eval/tree/main/benchmarks/industrial)
kills a process after a SQLite ledger write, preserves completed samples, and
distinguishes safe replay from duplicate writes. It uses three synthetic cases
per handler and no model API calls; it is not a production recovery guarantee.

## Evidence mapping

* Case IDs, full definitions, and manifest digests become native sample metadata.
* Text context is sent as a system message; conversation messages precede the
  current input. Multimodal content needs explicit native task configuration.
* Graders receive the actual native conversation and a projection of assistant
  tool calls and tool responses. An authored static trace cannot stand in for
  the actual execution trace.
* Skip values remain null in Inspect scores. Grader exceptions remain sample
  errors, rather than becoming failed quality judgments.
* Imported epochs are grouped by case. Each trial records the native log
  location, sample identity, sample digest, and model usage.
* Incomplete upstream runs carry report-level evidence issues and cannot pass
  a Multivon acceptance policy.

For a retry chain, call `from_inspect_log(final_log, previous_logs=[earlier_log])`
with earlier logs in chronological order. Preserved sample UUIDs are deduplicated;
new executions remain distinct. `trial_scope="final_attempt"` selects the last
attempt **per epoch** on imported Inspect evidence. The default all-attempts
policy includes interrupted attempts and therefore can remain indeterminate
after a successful retry. The bridge cannot discover omitted historical logs.

## Retry compatibility (development)

Native retry preservation by sample ID is not evidence that a task or grader
definition stayed unchanged. Use `bind_inspect_task` inside the registered task
factory, every time it is reconstructed:

```python theme={null}
from multivon_eval.integrations.inspect import bind_inspect_task

def build_declared_task(previous_log=None):
    native = Task(dataset=to_inspect_dataset(manifest), solver=fixture_answer(),
                  scorer=as_inspect_scorer(ExactMatch()), epochs=2)
    return bind_inspect_task(
        native, version="authorization-fixture/v1", dependencies={},
        configuration={"solver_behavior": "fixed hold response"},
        previous_log=previous_log,
    )

declared_log = inspect_eval(build_declared_task(), model="mockllm/model",
                            log_dir="inspect-declared-logs", display="none")[0]
# Reconstruct without running the solver; this rejects incompatible definitions.
rebuilt = build_declared_task(previous_log=declared_log)
assert rebuilt.metadata["multivon_task_contract_v1"]["digest"] == (
    declared_log.eval.metadata["multivon_task_contract_v1"]["digest"]
)
```

This local example does not retry a failed task. In an actual registered factory,
load the expected prior native log from a caller-selected path when reconstructing
for `eval_retry`. The [complete registered fixture and driver](https://github.com/multivon-ai/multivon-eval/blob/main/benchmarks/industrial/retry_contract_experiment.py)
demonstrate that wiring, native sample preservation, a rejected preflight and an
unguarded negative control. Binding only the first in-memory Task cannot guard
a later factory reconstruction.
The [frozen study and raw evidence](https://github.com/multivon-ai/multivon-eval/blob/main/benchmarks/industrial/RETRY_CONTRACT_VALIDATION.md)
record all three cases: native mixed scores pass 3/3 while fresh changed-rule
grading passes 2/3; the guarded retry adds zero target calls.

The function updates the Task and sample metadata in place. It reuses existing
grader/dependency fingerprints, engine inventory and named-file hashes; it does
not add a scheduler, checkpoint store or dataset format. Static datasets need
explicit unique sample IDs and bridge metadata. Graders need unique names and
`as_inspect_scorer`; custom graders also need `declare_dependencies`.

Use `files={"policy": policy_path, "task_source": source_path}` for immutable
policy/code/configuration inputs that must be rehashed. Use `dependencies` for
external revision identifiers. Keep runtime state such as ledger rows separate
from immutable code/configuration inputs. The caller's `version` and
`configuration` must cover hidden solver arguments, environment/sandbox image
revisions, model clients, services and closure state. A matching declaration is
an assertion about that state, not automatic discovery or proof of immutability.

The preflight compares the recorded contract before Inspect executes solvers.
Task construction and grader preparation occur before binding and may already
have side effects. Native `eval`/`eval_retry` overrides are not all available to
the factory; keep them in the declared configuration and supply the complete log
chain when importing. Native task/plan/execution changes across supplied logs
also make acceptance indeterminate. Logging-only settings may differ.

Each scorer records configuration before and after grading. Import checks sample
bindings, native input/reference identity, observed grader drift and mixed task
definitions. Compatibility issues remain in trial evidence through regrading.
An incompatible native retry can therefore be diagnosed even if the caller
omitted the preflight and Inspect already reused completed samples.

Legacy retry chains without declarations are still readable, but their compatibility
is unknown and cannot pass acceptance. Start a new declared evaluation; do not
retroactively invent contracts for historical logs. Digests detect accidental
changes, not forged evidence. Snapshots cannot detect transient changes restored
between observations or establish arbitrary in-memory/sandbox checkpoint safety.

Use [Inspect View](https://inspect.aisi.org.uk/log-viewer.html) for the full native
conversation and events. The adapter does not implement another generic log UI.

## Execution limits and completion evidence (development)

Use Inspect's [sample limits](https://inspect.aisi.org.uk/setting-limits.html)
and [concurrency controls](https://inspect.aisi.org.uk/parallelism.html) directly.
For example, the offline task above can run with explicit controls:

```python theme={null}
logs = inspect_eval(task, model="mockllm/model", log_dir="inspect-bounded-logs",
                    max_samples=2, max_connections=1, time_limit=60,
                    display="none", log_model_api=True)
report = from_inspect_log(logs[0])
```

These are example values, not production recommendations. `max_samples` bounds
simultaneous samples; `max_connections` bounds upstream model connections.
Inspect still scores the partial output after a sample limit stops execution.
A native log marked `success` therefore does not imply every task completed.

Development imports retain the native stop reason, invalidation, selected
resource/generation settings and measured durations under each trial's
`upstream.execution`. Unknown completion after a limit produces an infrastructure
error in the bridge and an indeterminate acceptance decision, while retaining
all grader scores. Native invalidations and errors also remain blocking evidence.
Regrading saved text cannot clear these execution constraints.

If the task was explicitly defined to score output at a token boundary, declare
that policy when importing:

```python theme={null}
bounded_report = from_inspect_log(logs[0], accepted_limits=("token",))
```

This declaration permits that limit type; it does not establish that the task
succeeded. Require independent outcome checks and apply the same declared policy
to the entire retry chain. Unknown limit types are rejected. A time, operator or
other stop remains blocking unless its type is also explicitly accepted; errors
and invalidations cannot be overridden with `accepted_limits`.
Increasing the general error budget does not authorize a new stop boundary or
restore invalidated measurements.

The offline [control experiment](https://github.com/multivon-ai/multivon-eval/blob/main/benchmarks/industrial/execution_controls_experiment.py)
uses Inspect's actual runtime and mock provider plus a real SQLite ledger.
It contrasts an acknowledgment with an independent persisted-state query, and
includes a completed positive control. Token and cost limits check observed usage;
an in-flight generation can exceed the threshold. The tested Inspect 0.3.263
`turn_limit=1` path made two mock generations before stopping. These controls must
not be presented as strict request reservations or hard monetary ceilings.
The [validation study and raw evidence](https://github.com/multivon-ai/multivon-eval/blob/main/benchmarks/industrial/EXECUTION_CONTROLS_VALIDATION.md)
retain all seven ledger scenarios, concurrency and cancellation results.

Cancellation stops cooperative async work and retains native cancellation/error
logs; it does not undo completed remote calls or SQLite commits. Configure
provider/tool timeouts and a safe replay policy for side effects. Worker threads
and code that blocks or suppresses cancellation require process/sandbox controls.

## Native offline runner controls (development)

`EvalSuite.run` accepts positive integer `workers` and `runs`; `workers=None`
selects the documented default. `run_async` also requires positive integer
`concurrency` and, when set, `evaluator_concurrency`. The latter bounds evaluators
**across the whole run**; it is not multiplied by the number of cases. Invalid
controls and nonfinite/out-of-range quality gates fail before preparation or calls.

On cancellation or an escaping child error, the async runner cancels and awaits
its owned async tasks without cancelling unrelated application tasks. Default
synchronous graders execute in threads: their awaits can be cancelled, but Python
cannot terminate the underlying thread. Use cooperative async graders/provider
timeouts or upstream sandbox isolation where stopping work is a requirement.
The native runner does not add durable resumption or a second deadline scheduler;
use Inspect for those execution facilities. Post-run `assert_budget` is a gate,
not an execution spend cap.

## Current boundaries

The importer requires samples and scorer metadata produced by this bridge.
Arbitrary native scorers need a deliberate score mapping; their numbers are not
silently treated as compatible Multivon verdicts. Native multimodal content and
event timing remain in Inspect's log; the text/tool projection is not lossless.

Multivon LLM graders still use their configured judge providers. Those calls
are **not automatically covered by Inspect model-call logging or its model
cost limits**. Use deterministic graders until you explicitly account for judge
requests and budgets. Imported sample errors are marked as infrastructure
errors without claiming to distinguish model errors from grader errors.

No average Inspect metric is registered by the scorer bridge: aggregating
nullable scores without required-check coverage would hide missing evidence.
Apply the [acceptance policy](/guides/acceptance-policies) after importing a log.