Skip to main content
Experimental in PyPI 0.19.0. Install with pip install 'multivon-eval[gymnasium]==0.19.0'. This adapter uses native Gymnasium environments and spaces. Use Inspect for durable execution and its existing sandbox providers for untrusted code. An agent can say “posted” without saving a record, save it twice, or alter a protected record and then restore it. Evaluate the required state and forbidden side effects separately from the final answer. A reward is useful evidence; it is not automatically your business success criterion.

​
Capture one task lifecycle

capture_episode creates an environment from your factory, resets it once, calls your interaction function, and attempts cleanup. Each independent repeat should receive a fresh environment and its own external resources. The helper does not provision a security sandbox or authenticate that isolation.
The factory and observer above are application-specific functions you supply. read_committed_state() takes no arguments and returns a portable JSON object. It should use a read-only database connection or state API, independently of the tool acknowledgement. Supply explicit environment/observer versions and portable options= for reset configuration. Version labels are caller assertions; they do not discover hidden implementation changes. The agent receives native Gymnasium observations and actions. Saved values use the spaces’ existing batched to_jsonable format, preserving numeric arrays without inventing another space codec. Recorded space descriptions are diagnostic; to decode values, use the same declared upstream spaces and from_jsonable. Reset/step info, observer state and options must be portable JSON. Unsupported values become capture errors rather than silently stringified data. The evidence retains the initial observation and independently read state, every attempted step, successful transitions, observations after exceptions, final state before cleanup, and cleanup status. An invalid action rejected before execution has no native action encoding; its error remains recorded. The original case identity stays bound to the episode. episode.data returns a detached copy; EpisodeEvidence.from_dict(...) verifies its digest and schema when restoring it.

​
Define outcome checks over saved evidence

Checks receive only saved evidence. They do not need to call the target again. Return True or False for a measured outcome and None when the required observation is missing. A raised exception is an evaluator error. Explicit check versions and recorded environment settings enter the report lock, so changing them blocks paired significance. Opaque callback behavior still requires review. Check forbidden side effects using the whole relevant history, not only final values. For example, inspect a database audit table or every captured state to detect a forbidden change followed by restoration. State snapshots between steps cannot detect an unrecorded transient change inside one step; use authoritative audit history when that distinction matters.

​
Completion, truncation and recovery

Gymnasium’s terminated means the environment reached a natural terminal state; it can represent failure, such as falling into a hole. truncated means execution ended outside that natural termination rule, such as reaching a time limit. The bridge preserves both flags. Returning early, truncation, observation errors, invalid transitions, or unconfirmed cleanup leave coverage issues. An otherwise passing report with these issues is indeterminate. A known measured failure can still reject under the acceptance policy even when other evidence is incomplete. Your check must decide which observations justify a verdict: a terminal-goal check should return None when a partial episode does not establish the answer. Do not treat an intermediate state as a completed task by accident. max_steps blocks further actions before calling the environment. This is a step bound, not a wall-clock timeout, memory limit or cancellation mechanism. A blocked synchronous callback cannot be interrupted by this helper. Ordinary exceptions are retained; cancellation such as KeyboardInterrupt propagates after cleanup, without promising a durable episode artifact. A factory that fails before returning an environment must clean up its own partial construction. Cleanup success means close() returned, not an independent resource-leak audit. A step can commit a write and then raise. The observer still runs, so that write is retained as partial-failure evidence. The bridge never automatically retries or assumes rollback. For recovery, preserve the failed attempt, deliberately reuse the intended persisted state, and apply an idempotency policy. Use Inspect when you need durable retry history, process-level deadlines and sandboxing. A passing recovery attempt does not erase or approve the failed history.

​
Run the real SQLite fixture

From the repository root:
The experiment reuses the document study’s posting handler. It tests correct and missing writes, idempotent and duplicate retries, forbidden changes followed by restoration, and explicit recovery after a committed write raises. It saves SQLite databases, episode evidence, reports, policy, protocol and hashes. There are no model calls. Native FrozenLake and CartPole checks also exercise discrete and array observations, terminal failures and truncation. The validation record contains the results and downloadable evidence. These are synthetic lifecycle checks, not production validation, a new benchmark dataset or a world-model quality result. World-model prediction and planning evaluation remain separate work in the implementation program.