> ## Documentation Index
> Fetch the complete documentation index at: https://opencompass-docs-preview-pr-335-0.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# agentcompass run

`agentcompass run` creates and executes one evaluation request with the selected benchmark, harness, model, and environment. `BENCHMARK HARNESS MODEL` are positional arguments in a fixed order; select the environment with `--env`.

```bash theme={"system"}
agentcompass run [OPTIONS] BENCHMARK HARNESS MODEL
```

## Run a Minimal Evaluation

The example uses [`sample_ids`](/en/user_guide/modules/benchmarks/overview#shared-benchmark-fields) in `--benchmark-params` to select one benchmark task by its stable task ID so you can quickly verify the component and endpoint configuration:

```bash theme={"system"}
export MODEL_NAME=""
export MODEL_BASE_URL=""
export MODEL_API_KEY=""

agentcompass run \
  swebench_verified \
  mini_swe_agent \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"
```

One `run` command corresponds to one evaluation request. To coordinate multiple explicit requests, use [`agentcompass launch`](/en/user_guide/using_agentcompass/cli/launch).

## Parameter Reference

This section explains how to use `agentcompass run` parameters and lists their complete signatures, defaults, and owners. See [Run Controls](/en/user_guide/using_agentcompass/run_controls) for guidance on concurrency, timeouts, retries, output, and debugging. “Built-in default” means the value used before user-level, project-level, or explicit configuration files override it. A conditional parameter is required only when the selected component or endpoint needs it.

### Component Selection and Parameters

| Parameter | Required? | Built-in default | What it controls |
| - | - | - | - |
| [`BENCHMARK`](/en/user_guide/modules/benchmarks/overview#find-a-benchmark) | Required | None | Registered benchmark ID. Determines dataset loading, task preparation, verification, and metrics. |
| [`HARNESS`](/en/user_guide/modules/harnesses/overview#find-a-harness) | Required | None | Registered harness ID. Determines the agent loop or framework used to attempt each task. |
| [`MODEL`](/en/user_guide/modules/models/overview#configure-the-model-spec) | Required | None | Primary model ID and the model-name segment used in the result path. Prefer `"$MODEL_NAME"` in shell commands. |
| [`--benchmark-params <json>`](/en/user_guide/modules/benchmarks/overview#configure-benchmark-parameters) | Conditional | Selected benchmark defaults | Overrides the shared and benchmark-specific fields defined by the selected benchmark config. |
| [`--harness-params <json>`](/en/user_guide/modules/harnesses/overview#configure-harness-parameters) | Conditional | Selected harness defaults | Overrides the complete parameter schema defined by the selected harness. |
| [`--model-base-url <url>`](/en/user_guide/modules/models/overview#configure-the-model-spec) | Conditional | `""` | Base URL of the model API that receives inference requests. |
| [`--model-api-key <key>`](/en/user_guide/modules/models/overview#configure-the-model-spec) | Conditional | `""` | Credential used to authenticate with the model API. |
| [`--model-api-protocol <protocol>`](/en/user_guide/modules/models/overview#model-api-protocol-list) | Optional | Unspecified | Sets the model API protocol. See the linked page for accepted values and selection rules. |
| [`--model-params <json>`](/en/user_guide/modules/models/overview#configure-the-params-field) | Optional | `{}` | Passes fields supported by the selected harness, API protocol, endpoint, and model deployment. |
| [`--env <id>`](/en/user_guide/modules/environments/overview#choose-a-provider) | Optional | `host_process` | Selects where task commands execute. Common choices are `docker`, `daytona`, and `modal`. |
| [`--env-params <json>`](/en/user_guide/modules/environments/configuration/overview) | Conditional | Selected environment and recipe defaults | Overrides the common Environment schema: setup, resources, network policies, and task environment variables. |

### Configuration and Recipes

| Parameter | Required? | Built-in default | What it controls |
| - | - | - | - |
| [`--config <path>`](/en/user_guide/using_agentcompass/cli/config) | Optional | None | Loads an additional YAML or JSON override file. Repeat to layer multiple files in command order. |
| [`--recipe <id>`](/en/user_guide/other_features/recipes#using-recipes) | Optional | All compatible built-ins | Restricts automatic recipe matching to the specified recipe IDs. Repeat to allow more than one. |
| [`--recipe-dir <path>`](/en/user_guide/other_features/recipes#trusted-external-recipes) | Optional | `[]` | Loads a trusted external recipe package directory. Repeat for multiple packages. |

### Execution Controls

| Parameter | Required? | Built-in default | What it controls |
| - | - | - | - |
| [`--execution-params <json>`](/en/user_guide/using_agentcompass/run_controls#per-task-phase-deadlines) | Optional | YAML execution defaults | Deep-merges common execution fields, including finalization grace, artifact preparation timeout, and artifact limits. Dedicated CLI options win. |
| [`--task-concurrency <int>`](/en/user_guide/using_agentcompass/run_controls#scale-concurrency-safely) | Optional | `32` | Limits concurrent physical attempt executions, including retries. |
| [`--max-retries <int>`](/en/user_guide/using_agentcompass/run_controls#retry-only-transient-failures) | Optional | `0` | Retries matching task or scoring failures up to this many times within the current logical attempt. |
| [`--retry-pattern-list <json>`](/en/user_guide/using_agentcompass/run_controls#retry-only-transient-failures) | Optional | `null` | Match ERROR message/code; null or \[] retries only FATAL. WARNING never retries. |
| [`--keep-environment`](/en/user_guide/using_agentcompass/run_controls#keep-environments-for-debugging) | Optional | Disabled | Skips environment cleanup so task and verifier sandboxes remain available for debugging. |

### Configure Repeated Attempts

| Parameter | Required? | Built-in default | What it controls |
| - | - | - | - |
| `--k <int>` | Optional | `1` | Sets the maximum independent attempts per task. |
| `--attempt-strategy <strategy>` | Optional | `avg` | Selects how repeated attempts execute and aggregate. |

For example, the following options plan three independent attempts per task and aggregate them with `avg`:

```bash theme={"system"}
agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --k 3 \
  --attempt-strategy avg
```

When these options are omitted, AgentCompass uses `k=1` and `avg`: each task runs once and its metrics use `native@1`. With `k>1`, attempts are independent execution units and share the `--task-concurrency` limit; the two strategies differ as follows:

| `strategy` | Compatible Benchmark primary | Execution and result |
| - | - | - |
| `avg` | Scalar `score` or binary `correct` | Plans all `k` attempts and computes `avg@k`; a binary metric also produces `pass@k`. |
| `pass` | Binary `correct` only | Stops after the first success or after at most `k` attempts; produces only the primary metric's `pass@k`. |

Keep these boundaries in mind:

* A scalar primary with `pass` raises an error before tasks start.
* `k` counts logical attempts and excludes retries created within an attempt by `--max-retries`.
* See [Metrics and Aggregation](/en/user_guide/other_features/results/metrics_aggregation) for exact `avg@k` and `pass@k` definitions and missing-attempt behavior.

With `launch`, each request can set its own `k` and strategy; see [Configure k and Strategy per Request](/en/user_guide/using_agentcompass/cli/launch#configure-k-and-strategy-per-request).

### Output and Reuse

| Parameter | Required? | Built-in default | What it controls |
| - | - | - | - |
| [`--run-name <name>`](/en/user_guide/using_agentcompass/run_controls#name-a-new-run) | Optional | `""` | Adds an optional namespace between `results_dir` and the combined Model/Benchmark/Harness directory. |
| [`--run-id <id>`](/en/user_guide/using_agentcompass/run_controls#name-a-new-run) | Optional | Current timestamp | Sets the final run-directory name instead of generating `YYYYMMDD_HHMMSS`. |
| [`--reuse [run-id]`](/en/user_guide/using_agentcompass/run_controls#resume-an-interrupted-run) | Optional | Disabled | Reuses compatible complete details without errors, or checkpoints for unfinished tasks without a complete detail, from the latest run under the same Model/Benchmark/Harness output namespace or a specified run ID. |
| `--no-checkpoint-resume` | Flag | Disabled | Do not use post-agent checkpoints to resume pending fresh evaluations. Complete results and terminal-attempt scheduling records keep their ordinary reuse behavior. |

### Process Settings

| Parameter | Required? | Built-in default | What it controls |
| - | - | - | - |
| [`--results-dir <path>`](/en/user_guide/using_agentcompass/run_controls#name-a-new-run) | Optional | `results` | Sets the root directory for run outputs. |
| [`--data-dir <path>`](/en/user_guide/other_features/results/overview#data-cache-and-output-directories) | Optional | `data` | Sets the root directory for downloaded datasets, caches, and prepared benchmark data. |
| [`--timeout-seconds <int>`](/en/user_guide/using_agentcompass/run_controls#set-an-appropriate-timeout) | Optional | `360000` | Sets the overall timeout in seconds for the evaluation execution phase after component preflight. Explicitly set `0` to disable this outer limit. Component-specific command and verifier timeouts remain separate. |
| [`--env-open-qps <provider=qps>`](/en/user_guide/using_agentcompass/run_controls#scale-concurrency-safely) | Optional | Local: `0`; remote: `10` | Limits environment creation rate per provider. Repeat for multiple providers; `0` disables pacing. |
| [`--provider-limit <provider=count>`](/en/user_guide/using_agentcompass/run_controls#scale-concurrency-safely) | Optional | `128` per built-in provider | Sets a process-wide provider limit on physical attempt executions, including retries. Repeat per provider; `0` disables the limit. |
| [`--progress <mode>`](/en/user_guide/using_agentcompass/run_controls#logs-and-progress) | Optional | `auto` | Selects terminal progress output: `auto`, `plain`, or `none`. |
| [`--log-level <level>`](/en/user_guide/using_agentcompass/run_controls#logs-and-progress) | Optional | `INFO` | Sets console logging to `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL`. |
| [`--file-log-level <level>`](/en/user_guide/using_agentcompass/run_controls#logs-and-progress) | Optional | `DEBUG` | Sets the run log-file level independently from console logging. |
| [`--auto-install-dependencies`](/en/user_guide/using_agentcompass/dependencies#automatic-installation) | Optional | Disabled | Allows AgentCompass to install missing trusted component extras in the active Python environment. |

### Analysis

| Parameter | Required? | Built-in default | What it controls |
| - | - | - | - |
| [`--enable-analysis`](/en/user_guide/using_agentcompass/cli/analysis#run-with-evaluation) | Optional | Enabled | Enables registered post-evaluation analyzers. Use `--no-enable-analysis` to skip them. |
| [`--analysis-params <json>`](/en/user_guide/using_agentcompass/cli/analysis#run-with-evaluation) | Optional | Built-in analyzer defaults | Selects analyzers and overrides their concurrency, model, rendering, or analyzer-specific settings. |

### Component-Specific JSON Parameters

The four JSON parameter flags do not share one schema. Their available fields and defaults depend on the selected
component:

| Flag | Owner | Where to find its fields |
| - | - | - |
| `--model-params` | Harness, API protocol, and endpoint | [Model parameter schema](/en/user_guide/modules/models/overview#configure-the-params-field) and the endpoint provider's request schema |
| `--benchmark-params` | Selected benchmark | [Benchmark parameter schema](/en/user_guide/modules/benchmarks/overview#configure-benchmark-parameters) and `agentcompass config docs benchmark <id>` |
| `--harness-params` | Selected harness | [Harness parameter schema](/en/user_guide/modules/harnesses/overview#configure-harness-parameters) and `agentcompass config docs harness <id>` |
| `--env-params` | Selected environment | [Environment parameter schema](/en/user_guide/modules/environments/configuration/overview) and `agentcompass config docs env <id>` |

[`sample_ids`](/en/user_guide/modules/benchmarks/overview#shared-benchmark-fields) belongs in `--benchmark-params`; repeated attempts use the execution controls above. Provider CPU, memory, image, and network settings belong in `--env-params`. See [Configure an Evaluation](/en/user_guide/using_agentcompass/overview#evaluation-structure) for the ownership map.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.