agentcompass launch

Use a YAML or JSON orchestration file to coordinate multiple explicit evaluation requests with one global scheduler.

agentcompass launch <orchestration.yaml> [OPTIONS]

Each request in the orchestration still selects one benchmark, harness, model, and environment, using the same structure as an evaluation request created by agentcompass run. Use run for one request. Use launch to compare models, evaluate several benchmarks, or mix harnesses and environments across multiple requests. AgentCompass does not infer a matrix. Every request is named and declared explicitly, which keeps its parameters, results, failures, and reuse source auditable.

Define an Orchestration

The following orchestration defines two evaluation requests. They share one global pool of 16 physical attempt-execution slots. Common model settings are defined once under defaults, while each request selects its own k and aggregation strategy:

# terminal-evaluations.yaml

# Maximum physical attempt executions across all requests, including retries.
task_concurrency: 16

# Values inherited by every request unless that request overrides them.
defaults:
  model:
    id: ${MODEL_NAME}
    base_url: ${MODEL_BASE_URL}
    api_key: ${MODEL_API_KEY}
    api_protocol: openai-chat
    params:
      temperature: 1
      top_p: 0.95

requests:
  # A user-defined request name used for its result directory, progress, and outcomes.
  - name: tb2vrf
    benchmark:
      id: terminal_bench_2_verified
    harness:
      id: terminus2
      max_turns: 300
    environment:
      id: docker
    execution:
      attempts:
        k: 3
        strategy: avg

  - name: tb21
    benchmark:
      id: terminal_bench_2_1
    harness:
      id: terminus2
      max_turns: 300
    environment:
      id: daytona
    execution:
      attempts:
        k: 5
        strategy: pass

Export every referenced environment variable before resolving the file:

export MODEL_NAME=""
export MODEL_BASE_URL=""
export MODEL_API_KEY=""

Environment references must occupy the complete field, as in ${MODEL_API_KEY}. AgentCompass rejects partial string interpolation so unresolved or accidentally concatenated secrets do not silently enter a request.

What the fields mean

Field	Meaning
`task_concurrency`	One global physical attempt-execution limit shared by every request; retries consume the same pool. It is not applied independently to each request.
`defaults`	Values inherited by all requests. A request may override only the fields that differ.
`defaults.model.id`	The actual model ID sent to the endpoint and recorded in result metadata.
`base_url` / `api_key` / `api_protocol`	Connection settings for the shared model endpoint. Environment references keep credentials out of the YAML file.
`model.params`	Model inference parameters forwarded through the selected harness/protocol, such as `temperature` and `top_p`.
`requests`	Evaluation requests to schedule; declaration order determines scheduling priority.
`requests[].name`	A unique, user-defined request name used for its result directory, progress display, and request-level outcomes. `benchmark.id` selects the benchmark.
`requests[].execution.attempts.k`	Attempts per task for this request. It must be a positive integer and defaults to `1`.
`requests[].execution.attempts.strategy`	Aggregation strategy for this request: `avg` or `pass`, defaulting to `avg`; scalar primary metrics support only `avg`.
`benchmark.id`	The real registered benchmark ID, such as `terminal_bench_2_1`.
`harness.id`	The real registered harness ID, such as `terminus2`.
`harness.max_turns`	A harness-specific parameter. Component fields are written beside `id`, without a `params` wrapper.
`environment.id`	The real registered Environment provider ID, such as `daytona` or `docker`.

The request name therefore remains stable even if its benchmark or environment configuration changes. Use agentcompass list benchmark, agentcompass list harness, and agentcompass list env to inspect valid component IDs.

Mapping rules

Section	Shape	Meaning
`model`	`id`, endpoint fields, and optional `params`	Generation and provider request options remain under `model.params`.
`benchmark`, `harness`, `environment`	`id` plus component fields at the same level	`id` selects the component; every other field becomes one of its parameters. Do not add a `params` wrapper.
`execution`	Partial execution mapping	Controls repeated attempts, analysis, retries, recipes, and environment retention for the request. The orchestration still owns global task concurrency.
Request `runtime`	`reuse`, `reuse_run_id`, and `checkpoint_resume`	Selects an earlier run and whether pending fresh evaluations can resume from post-agent checkpoints.
`output`	`run_name` and `run_id`	Organizes the new request result directory.

The optional version defaults to the latest supported orchestration format. Request names must be non-empty and unique. AgentCompass normalizes each name to one safe directory component and rejects names that resolve to the same output namespace, even when their run IDs differ. Each request writes results under:

<results_dir>/[<output.run_name>/]<requests[].name>/<run_id>/

For the example above, --run-id baseline creates these two directories under the default result root:

results/tb2vrf/baseline/
results/tb21/baseline/

output.run_name adds an optional grouping prefix. Distinct request names keep outputs separate, including when requests use the same Model and Benchmark with different Harnesses. An explicit run ID must be unused within its request namespace.

Configure k and strategy per request

launch does not use one --k or --attempt-strategy override for every benchmark. As shown above, put attempts under requests[].execution to select k and strategy independently for each request; see agentcompass run’s “Configure Repeated Attempts” for the complete values and execution behavior. AgentCompass resolves each request independently from the shared configuration and defaults.execution, then applies that request’s own execution fields. Requests do not inherit from or overwrite one another. Even if a later request starts while an earlier request is still running, it cannot change the earlier request’s resolved k or strategy. The requests share only the execution slots defined by the top-level task_concurrency. When every request uses the same settings, you can put attempts under defaults.execution; any request can still override them. For an orchestration that mixes scalar and binary primary metrics, set strategy explicitly in each request so that a scalar benchmark cannot inherit an incompatible pass strategy. See Metrics and Aggregation for metric types, strategy constraints, and result semantics.

Validate Before Running

Resolve the complete orchestration before starting an evaluation:

agentcompass launch terminal-evaluations.yaml --dry-run

--dry-run loads configuration layers, expands environment references, resolves component defaults, validates every request, and prints a redacted orchestration. It does not load benchmark tasks or create result directories. Review the selected component IDs, task filters, environments, endpoint hostnames, concurrency, and reuse settings in this output. Start the same orchestration after validation:

agentcompass launch terminal-evaluations.yaml

For a one-off change, CLI options can override the shared run controls in the orchestration file:

agentcompass launch terminal-evaluations.yaml \
  --task-concurrency 8 \
  --provider-limit docker=8 \
  --progress plain

Use agentcompass launch --help for the complete option list. Common orchestration-level options include:

Option	Purpose
`--task-concurrency <n>`	Sets the physical attempt-execution limit shared by every request, including retries.
`--timeout-seconds <n>`	Sets one wall-clock deadline for the complete orchestration. The default is `360000` seconds (100 hours); explicitly set `0` to disable it.
`--provider-limit <provider>=<n>`	Caps simultaneous attempts using a provider; repeat for multiple providers.
`--env-open-qps <provider>=<qps>`	Limits environment startup rate; repeat for multiple providers.
`--progress auto\|plain\|none`	Selects the multi-request terminal renderer.
`--auto-install-dependencies`	Allows trusted missing host-side extras to be installed; disabled by default.
`--reuse`	Enables latest-run reuse within each request’s output namespace by default.
`--run-id <id>`	Applies one explicit run ID to every new request output; distinct request namespaces can use the same ID.
`--dry-run`	Resolves, validates, redacts, and prints without executing.

Understand Scheduling and Failure Isolation

All requests share one execution worker pool. Declaration order defines admission priority: tasks from an earlier request are admitted first, and later requests use idle slots after all pending tasks from earlier requests have been admitted. This ordering is deterministic, but it does not force one complete evaluation to finish before the next begins. In the task_concurrency: 16 example under Define an Orchestration:

AgentCompass fills available slots with tasks from tb21 first.
As tb21 tasks finish, its remaining unstarted tasks continue to receive priority.
Once all tb21 tasks have been admitted, any free slots immediately begin tb2vrf tasks, even if the final tb21 tasks are still running.
If tb21 contains fewer than 16 tasks, the unused slots begin tb2vrf immediately.

This is ordered admission with overlap, not a strict barrier between requests. Request order controls which pending tasks get capacity first; task_concurrency controls concurrent physical attempt executions across the orchestration, including retries and repeated attempts from the same task. Each request keeps its own run directory, progress files, logs, summary, and terminal outcome. A request-level failure is recorded as failed and does not prevent later requests from running. The orchestration returns completed when all requests complete, partial_failure when only some fail, and a terminal timeout or cancellation status when the shared operation is stopped. See Logs and Progress for the terminal behavior of all three progress modes and how they interact with progress files. Use the capacity guidance in Run Controls before raising global concurrency.

Reuse Existing Runs

--reuse enables latest-run reuse by default for every request in the orchestration:

agentcompass launch terminal-evaluations.yaml --reuse

An individual request can opt out with runtime.reuse: false. To select an exact source, set runtime.reuse_run_id in that request or in defaults; output.run_id names the new result and is not a reuse source. Reuse searches only within the current request’s result namespace, including its optional output.run_name prefix. Requests with distinct normalized names can independently reuse their latest runs even when their Model and Benchmark are the same. The selected source must have the same Benchmark ID and attempt plan; evaluation settings can change. Changing a request’s name changes where reuse looks. Existing result directories remain in place. Automatic reuse searches only the new request-based hierarchy; it does not discover runs in the former Benchmark/Model hierarchy. Conflicting normalized request namespaces and existing explicit output directories are rejected before task execution. See Resume an Interrupted Run for reuse matching and usage constraints.

Using AgentCompass

Modules

Other Features

Define an Orchestration

What the fields mean

Mapping rules

Configure k and strategy per request

Validate Before Running

Understand Scheduling and Failure Isolation

Reuse Existing Runs

​Define an Orchestration

​What the fields mean

​Mapping rules

​Configure k and strategy per request

​Validate Before Running

​Understand Scheduling and Failure Isolation

​Reuse Existing Runs

​Related Pages

Define an Orchestration

What the fields mean

Mapping rules

Configure k and strategy per request

Validate Before Running

Understand Scheduling and Failure Isolation

Reuse Existing Runs

Related Pages