Skip to main content
AgentCompass uses one metric pipeline for binary, scalar, and mixed Benchmarks. The Benchmark declares what each attempt measures; the run configuration selects how repeated attempts are executed and reduced; the result report keeps a separate value and coverage count for every metric series.

​
Configure Repeated Attempts

Repeated attempts are execution controls, not Benchmark parameters. Set them with CLI options or under execution.attempts in a configuration file:

​
Understand Metric Contracts

Each Benchmark declares a Metric Contract. It assigns every key in attempts.<N>.metrics one of two kinds: binary_success means a yes/no success condition defined by the Benchmark, such as whether a verifier passed. It is not simply any numeric field whose current values happen to be 0 and 1. A scalar represents an amount or degree, including partial credit. Every contract declares exactly one primary metric. Binary primaries use the canonical ID correct; scalar primaries use score; neither canonical ID can be auxiliary. Benchmark-specific names such as reward and f2p are auxiliary metrics. A mixed Benchmark can declare both binary and scalar observations, but its fixed primary metric always controls the execution strategy. The contract is validated before tasks run. Selecting strategy: pass for a Benchmark with a scalar primary raises an error, even at k=1, because a numeric score does not define success. Only Benchmarks whose primary metric is correct can use pass.

​
Know Which Series Are Produced

There is no attempt-1 or first headline when k>1. For an avg run with a binary primary, both its avg@k and pass@k are headline results. Other contract metrics are retained as auxiliary series in the full report. The reducer definitions are:
  • native@1: the single valid observation.
  • avg@k: the arithmetic mean of exactly k valid observations. For a binary metric, true is 1 and false is 0.
  • pass@k: 1 as soon as any valid binary observation is true; 0 only after all k observations are valid and false.

​
Treat Failed and Missing Attempts Explicitly

Final FATAL in any logical attempt invalidates the whole task. ERROR and WARNING preserve valid Benchmark observations; no missing metric is filled with false/0 by the reducer.
  • avg@k is unavailable unless all k observations are valid.
  • pass@k=1 is exact once a success exists, even if later attempts were not needed.
  • pass@k=0 is exact only when all k valid observations are false.
Counts include total, evaluated, unavailable, invalidated and diagnostic error. evaluated + unavailable + invalidated equals total. INVALIDATED has no value. ERROR means missing required attempts or final ERROR without a reducible value; UNAVAILABLE means insufficient observations without those errors.

​
Execution, Retry, and Reuse

execution.task_concurrency is the single per-run concurrency limit. It counts physical attempt executions, including retries, rather than treating all k attempts for one task as one slot. Inline analysis enabled by agentcompass run shares this limit; the standalone agentcompass analysis command schedules work with its own task concurrency. With strategy: avg, attempts from the same task may run concurrently only when both the Benchmark and Harness declare that their per-attempt state is isolated. Otherwise AgentCompass runs those attempts serially. The user-facing concurrency setting does not change. Reuse is decided per logical attempt under the current plan. Resolved failures remain history, old retry counts do not consume the new budget, and none/fresh evaluation snapshots can avoid repeating inference.

​
Aggregate Tasks and Categories

The repeated-attempt reducer and the Benchmark aggregator solve different problems. A reducer combines the k observations of one task; after that, the runtime calls Benchmark.aggregate_metrics() to apply the Benchmark’s official cross-task definition. Every derived metric excludes the same invalidated task IDs before computing its numerator, denominator and weights. Fixed-baseline medals keep their baseline denominator and are unavailable when selected applicable tasks lack observations. The default Benchmark implementation applies the following shared policies separately to every series: Every category and hierarchy node stores the same four series-specific counts as the overall value. Missing children have value: null and do not borrow another series’ count. For hierarchy nodes, unweighted, explicit weighted, and weighted_by_count aggregation renormalize over children with valid values. A Benchmark overrides the default hook when its official result is not a mean of task values. For example, SciCode computes subproblem accuracy as sum(correct subproblems) / sum(total subproblems), DeepResearch FACT weights citation accuracy by checked citations, GDPVal divides the corpus total score by the corpus maximum, and Frontier Engineering derives medal and rank results from its reference tables. These formulas run after the selected task-level reducer, so they remain compatible with native@1 and complete avg@k observations. In metrics.json, each series records its actual aggregation formula. Shared series use micro_weighted, category_mean, or category_hierarchy; custom series can use ratio_of_sums, sum, or benchmark. Formula inputs and totals are stored in series[].extra, while larger Benchmark-specific diagnostics such as rank comparisons are stored in the report-level extra. Category and hierarchy breakdowns also expose aggregation_weight when a formula uses a denominator other than the number of evaluated tasks.

​
Read the Outputs

Successful aggregation writes two complementary files: The CLI also prints the headline series. Use metrics.json for tooling and audits; do not parse Markdown as the data source. See Task Results for attempt observations and Summary and Analysis Results for the complete output layout.