SWE-bench Pro Verified

SWE-bench Pro Verified is a verified version of SWE-bench Pro that addresses two sources of evaluation unreliability identified through trajectory analysis: reward hacking caused by leakage of gold solutions or hidden evaluation information, and task quality issues such as misleading problem statements and improperly scoped tests (paper, dataset, evaluation scripts). The benchmark contains 731 tasks. Four anti-hacking controls, including repository reconstruction, test artifact concealment, metadata filtering and anonymization, and network blocking, apply to all tasks; task refinement further corrects 102 tasks with confirmed quality issues: 22 misleading descriptions, 75 overly narrow tests, 3 overly broad tests, and 2 other issues.

How it works

A task has separate inference and evaluation stages:

Load and prepare. AgentCompass reads instance_id, repository, base commit, problem statement, requirements, and any newly introduced interface from the dataset. A provider recipe normally selects the task’s prebaked image and exposes its repository at /app.
Apply the refined data. AgentCompass loads the complete task data from swebench_pro_verified.jsonl in opencompass/SWEBench-Pro-Verified.
Apply anti-hacking controls. AgentCompass removes evaluation test files, rebuilds the repository as a fresh commit with its Git history erased, blocks code-hosting domains through a blacklist, and filters and anonymizes task metadata such as instance_id.
Run the coding agent. A harness such as mini-SWE-agent or OpenHands receives the issue and edits the checked-out repository. It must write the final unified diff to /app/patch.txt when using the standard recipe layout.
Start a fresh evaluation environment. Inference changes are not trusted as the evaluation workspace. AgentCompass starts a new environment from the task image, resets /app to base_commit, and applies the patch.
Run the official instance scripts. The benchmark loads the task’s run_script.sh and parser.py from the local run_scripts/<instance_id>/ tree, downloading missing scripts from SWE-bench_Pro-os. The parser turns test logs into structured results.
Decide resolution. A task is resolved=true only when every required FAIL_TO_PASS and PASS_TO_PASS test appears in the passed-test set.

Environments

This benchmark currently cannot run on Daytona or Modal because neither provider supports blacklists; they support only whitelists or complete outbound-network blocking (Daytona network limits, Modal sandbox networking). Docker is currently the only public provider that can enforce a blacklist; see Network Configuration. Each task environment requires at least 4 CPU cores and 8 GiB of memory. Any Environment provider applies these two values by default. These values can be overridden manually, but allocating fewer resources than required is not recommended; see Resource Limits.

Parameters

Pass benchmark configuration via --benchmark-params '{...}', or through benchmark.params in a YAML file given to --config; the CLI wins on shared keys.

Parameter reference

Parameter	Type	Default	Choices / values	Description
`prepare_mode`	string	`git_clone`	`git_clone` / `prebaked`	How the inference repository is prepared. Built-in provider recipes normally replace this with `prebaked`.
`workspace_root`	string	`/app`	absolute environment path	Root used for task workspaces before recipe overrides.
`repo_url_template`	string	`https://github.com/{repo}.git`	template containing `{repo}`	Repository clone URL used in `git_clone` mode.
`scripts_dir`	string	`""`	local directory	Controller-side directory containing `<instance_id>/run_script.sh` and `parser.py`. Empty resolves to the data directory’s `run_scripts/`.
`dockerfiles_dir`	string	`""`	local directory	Controller-side official Dockerfile root used to recover task environment exports. Empty resolves under the data directory.
`evaluation_repo_dir`	string	`/app`	absolute environment path	Repository path in the evaluation image; recipes keep it at `/app`.
`evaluation_workspace_dir`	string	`/app`	absolute environment path	Directory where the patch, scripts, logs, and parser output are staged during evaluation.
`sample_ids`	list / string / null	`null`	valid instance ids	Optional exact task filter. Unknown ids fail fast.

The model id is the third positional argument to agentcompass run, not a --benchmark-params field. This benchmark does not expose a split parameter: it loads the public test split.

Inference, model, and evaluation controls

What is limited	mini-SWE-agent	OpenHands	SWE-bench Pro Verified
One model request	`--model-params.timeout` (unset by AgentCompass)	`--model-params.timeout`, otherwise `conversation_timeout=3600`	—
One repository command	`command_timeout=2400`	`command_timeout=1800`; no-change soft limit `600`	—
Agent loop	`step_limit=250`, `cost_limit=3.0`	`max_iterations=250`	—
Whole inference task	`--execution-params.run_timeout_seconds=null`	`--execution-params.run_timeout_seconds=9600`	—
Fresh official evaluation	—	—	`--execution-params.evaluation_timeout_seconds=3600`
Repeated attempts	—	—	`--k`, `--attempt-strategy`

evaluation_timeout_seconds controls only the fresh run_script.sh and parser evaluation after patch collection. It cannot extend inference. Thinking/reasoning belongs in --model-params; use the protocol/provider form documented for mini-SWE-agent or OpenHands.

Run examples

agentcompass run takes three positional arguments in order: Benchmark, Harness, and Model. The examples use swebench_pro_verified; harness choices are described below. Before running, make sure local Docker is available and set MODEL_NAME, MODEL_BASE_URL, and MODEL_API_KEY to the model under test, API endpoint, and API key.

Recommended harness

mini-SWE-agent is the recommended harness for SWE-bench Pro. It uses the benchmark-specific mini-SWE-agent configuration and executes repository commands in the task environment.

Smoke test (single task end-to-end)
Custom parameters
AgentCompass recommended config

Run one task to verify inference, patch collection, and official evaluation end to end.

agentcompass run \
  swebench_pro_verified \
  mini_swe_agent \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "sample_ids": ["instance_NodeBB__NodeBB-04998908ba6721d64eba79ae3b65a351dcfbc5b5-vnan"]
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Run three attempts for one task and customize the attempt policy, model request, command, task, and evaluation limits.

agentcompass run \
  swebench_pro_verified \
  mini_swe_agent \
  "$MODEL_NAME" \
  --env docker \
  --k 3 \
  --attempt-strategy pass \
  --benchmark-params '{
    "sample_ids": ["instance_NodeBB__NodeBB-04998908ba6721d64eba79ae3b65a351dcfbc5b5-vnan"]
  }' \
  --execution-params '{
    "evaluation_timeout_seconds": 4800,
    "run_timeout_seconds": 14400
  }' \
  --harness-params '{
    "step_limit": 300,
    "cost_limit": 5.0,
    "command_timeout": 2400
  }' \
  --model-params '{
    "temperature": 0,
    "max_tokens": 32768,
    "timeout": 3600,
    "reasoning_effort": "high"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Evaluate the complete public split with explicit inference and evaluation limits. Adjust --task-concurrency only when required by provider capacity.

agentcompass run \
  swebench_pro_verified \
  mini_swe_agent \
  "$MODEL_NAME" \
  --env docker \
  --execution-params '{
    "evaluation_timeout_seconds": 3600,
    "run_timeout_seconds": 12000
  }' \
  --harness-params '{
    "step_limit": 250,
    "cost_limit": 3.0,
    "command_timeout": 2400
  }' \
  --model-params '{
    "temperature": 0,
    "max_tokens": 32768,
    "timeout": 3600,
    "reasoning_effort": "high"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --task-concurrency 16

Other optional harnesses

OpenHands is also supported. The following command evaluates the full dataset and exposes its independent model-request, terminal-command, agent-loop, whole-task, and evaluation limits:

agentcompass run \
  swebench_pro_verified \
  openhands \
  "$MODEL_NAME" \
  --env docker \
  --execution-params '{
    "evaluation_timeout_seconds": 3600,
    "run_timeout_seconds": 12000
  }' \
  --harness-params '{
    "max_iterations": 250,
    "conversation_timeout": 3600,
    "command_timeout": 1800,
    "terminal_no_change_timeout_seconds": 600
  }' \
  --model-params '{
    "temperature": 0,
    "max_output_tokens": 32768,
    "timeout": 3600,
    "reasoning_effort": "high",
    "num_retries": 10,
    "retry_min_wait": 8,
    "retry_max_wait": 64,
    "retry_multiplier": 2
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Evaluation Results

For shared result conventions, see Run Directory, Aggregate Scores, and Task Files and Shared Fields.

Scoring Metrics

SWE-bench Pro Verified’s primary metric is binary correct, matching the evaluator’s resolved decision. Resolution follows the test rules in How It Works; there is no partial credit. With the default configuration, each task has one attempt and the overall score is the issue resolution rate over tasks with valid scores, ranging from 0 to 1; higher is better. See Metrics and Aggregation for repeated attempts, category aggregation, and scoring failure rules.

Task Results and Scoring Evidence

final_answer contains the unified diff patch submitted to the evaluator. eval_raw_data under meta.benchmark preserves scoring evidence, with these fields when available:

Field	Contents
`resolved`	The evaluator’s issue resolution decision.
`completed`	Whether the evaluator completed its decision.
`fail_to_pass`, `pass_to_pass`	Required fix tests and regression tests.
`fail_to_pass_missing`, `pass_to_pass_missing`	Required tests absent from the passing set, identifying why an issue remains unresolved.
`tests`, `raw_output`	Parsed test list and complete parsed output.
`stdout`, `stderr`, `returncode`	Test logs and the evaluation process exit code.
`error`, `timed_out`	Diagnostics for patch application or evaluation problems, present only on the relevant paths.

Using AgentCompass

Modules

Other Features

How it works

Environments

Parameters

Parameter reference

Inference, model, and evaluation controls

Run examples

Recommended harness

Other optional harnesses

Evaluation Results

Scoring Metrics

Task Results and Scoring Evidence

​How it works

​Environments

​Parameters

​Parameter reference

​Inference, model, and evaluation controls

​Run examples

​Recommended harness

​Other optional harnesses

​Evaluation Results

​Scoring Metrics

​Task Results and Scoring Evidence

How it works

Environments

Parameters

Parameter reference

Inference, model, and evaluation controls

Run examples

Recommended harness

Other optional harnesses

Evaluation Results

Scoring Metrics

Task Results and Scoring Evidence