> ## Documentation Index
> Fetch the complete documentation index at: https://opencompass-docs-preview-pr-335-0.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation Modes and Artifacts

Choose `none`, `reuse`, or `fresh` from the state the verifier must access, and extract any artifacts needed later before the task Environment closes.

## Three Modes

| Mode | <span style={{ display: "inline-block", minWidth: "14rem" }}><code>env</code> passed to <code>evaluate()</code></span> | Lifecycle | Use it when |
| - | - | - | - |
| `none` | `None` | Close the task Environment after execution and artifact collection, then evaluate in the AgentCompass process | Exact match, structured answers, or controller-side evaluators |
| `reuse` | The current task `EnvironmentSession` | Close the task Environment only after execution, artifact collection, and evaluation finish | The verifier must inspect the same filesystem or service state left by execution |
| `fresh` | A new evaluation `EnvironmentSession` | Extract artifacts and close the task Environment, then open a separate evaluation Environment | Evaluation needs isolated tests, a clean image, or a different network policy |

In all three modes, runtime executes declared collection commands while the environment is alive; without commands it collects existing files directly. Collection follows the resolved path and command lists, using Benchmark defaults unless execution parameters replace them. `execution.save_artifacts` controls local storage; fresh requires it to be enabled. Artifact preparation and collection do not decide the score.

## Set Defaults and Per-Task Overrides

Set a class-level default when every sample uses the same mode:

```python theme={"system"}
class ExampleBenchmark(BaseBenchmark):
    evaluation_environment_mode = "reuse"
```

Set the mode on individual `TaskSpec` values when samples require different verifier paths:

```python theme={"system"}
TaskSpec(
    task_id=task_id,
    question=question,
    category=category,
    ground_truth=ground_truth,
    evaluation_environment_mode="fresh",
)
```

The runtime Planner initially resolves the mode in this order:

```text theme={"system"}
RunRequest.environment.evaluation_environment_mode
  > TaskSpec.evaluation_environment_mode
  > Benchmark.resolve_evaluation_environment_mode(req)
```

This is the selection order while building the initial `ExecutionPlan`. Matching Recipes can then adjust the plan according to their own contracts. If a Recipe changes the evaluation Environment, its implementation and tests must state which fields it may override and which it must preserve. When debugging the effective mode, inspect `resolved_execution_plans` in `run_info.json`; it records the plan after Recipe adjustments.

## `none`: Evaluate in the AgentCompass Process

Use `none` when evaluation depends only on `TaskSpec`, `PreparedTask`, and `RunResult`. Do not read the task workspace in this mode:

```python theme={"system"}
async def evaluate(
    self,
    task: TaskSpec,
    prepared: PreparedTask,
    result: RunResult,
    req: RunRequest,
    plan: ExecutionPlan,
    env: EnvironmentSession | None = None,
) -> RunResult:
    _ = task, prepared, req, plan
    if env is not None:
        raise RuntimeError("none evaluation must not receive an Environment")
    return score_answer(result)
```

If a controller-side evaluator calls another Model, configure that judge Model explicitly and record its version and inference parameters. Do not silently let the Model under test score its own result.

## `reuse`: Inspect the Task Environment

`reuse` evaluates before the task Environment closes, so the verifier can see the workspace left by the agent:

```python theme={"system"}
class WorkspaceBenchmark(BaseBenchmark):
    evaluation_environment_mode = "reuse"

    async def evaluate(
        self,
        task: TaskSpec,
        prepared: PreparedTask,
        result: RunResult,
        req: RunRequest,
        plan: ExecutionPlan,
        env: EnvironmentSession | None = None,
    ) -> RunResult:
        _ = task, prepared, req, plan
        if env is None:
            raise RuntimeError("workspace verification requires reuse mode")
        verifier = await env.exec(
            ["python3", "/opt/verifier/check.py"],
            timeout=300,
        )
        return apply_verifier_result(result, verifier)
```

The verifier path and its dependencies must exist in the task Environment. Evaluation uses the resolved `evaluation_network_policy`; do not assume the execution-stage network permissions remain active.

For a complete production `reuse` implementation, inspect [`terminalbench2.py`](https://github.com/open-compass/AgentCompass/blob/main/src/agentcompass/benchmarks/terminalbench2/terminalbench2.py).

## `fresh`: Prepare, Collect, Then Verify in Isolation

`fresh` does not copy the entire workspace. Declare the files or directories needed by the verifier; shared runtime code collects them before closing the agent environment and restores them at the same absolute paths in the fresh environment:

```python theme={"system"}
TaskSpec(
    task_id=task_id,
    question=question,
    category=category,
    ground_truth=ground_truth,
    evaluation_environment_mode="fresh",
    artifacts=["/app/submission.patch"],
)
```

If the agent already writes these files, no preparation commands are needed. Otherwise declare `TaskSpec.artifact_collect` commands; Harbor's `verifier.collect` maps to the same contract. For example:

```python theme={"system"}
from agentcompass.runtime.artifacts import ArtifactCollectCommand

TaskSpec(
    task_id=task_id,
    question=question,
    category=category,
    ground_truth=ground_truth,
    evaluation_environment_mode="fresh",
    artifacts=["/app/submission.patch"],
    artifact_collect=[
        ArtifactCollectCommand("bash /opt/task/export-submission.sh", timeout_seconds=300),
    ],
)
```

The evaluator receives an environment with the declared files already restored. Do not download or upload them again:

```python theme={"system"}
async def evaluate(
    self,
    task: TaskSpec,
    prepared: PreparedTask,
    result: RunResult,
    req: RunRequest,
    plan: ExecutionPlan,
    env: EnvironmentSession | None = None,
) -> RunResult:
    if env is None:
        raise RuntimeError("isolated verification requires fresh mode")
    verifier = await env.exec(
        ["bash", "/opt/verifier/run.sh", "/app/submission.patch"],
        timeout=600,
    )
    return apply_verifier_result(result, verifier)
```

Preparation and collection have independent execution budgets. Failed preparation or transport retains diagnostics but does not proceed to grading or publish a resumable checkpoint; a normally missing submission remains a verifier scoring decision. See [Shared Contracts](/en/developer_guide/extensions/benchmark/code_implementation/shared_contracts#submission-artifacts-and-replay) for size bounds, integrity checks, and preparation-hook responsibilities. DeepSWE is a production example of preparation plus shared transfer.

## Check Before Choosing a Mode

* Choose `none` when scoring depends only on an answer or in-memory objects; do not create an extra sandbox for simple scoring.
* Choose `reuse` when the verifier must see the original filesystem modified by the agent, and ensure the verifier does not corrupt results that must be retained.
* Choose `fresh` when the verifier must not trust dependencies or processes left by the agent, and transfer only the minimum required submission.
* Set explicit timeouts for commands in `prepare_task()`, `artifact_collect`, and the verifier, and make every step safe to repeat during retries.
* Use different statuses for evaluation failure and a valid zero score; see [Results and Aggregation](/en/developer_guide/extensions/benchmark/code_implementation/results_and_aggregation) for the mapping.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.