DeepResearch Bench

DeepResearch Bench (arxiv) evaluates a deep-research agent’s ability to write research reports: given an open-ended research query that requires web search and multi-step evidence gathering, the agent produces a complete Markdown research report, which is then graded by two frameworks — RACE ** for report quality and ** FACT ** for citation grounding. The dataset contains ** 100 tasks (50 Chinese, 50 English), written by domain experts and spanning 22 topics.

How it works

A DeepResearch Bench run has two stages — inference and scoring. The scoring stage comprises two mutually independent frameworks, RACE and FACT; use metrics to select one or both.

Inference and scoring

Inference. The model under test acts as a research agent and, driven by the harness (default naive_search_agent), completes multi-turn tool loops such as search / visit per task, ultimately producing a Markdown research report as its answer for that task.
Scoring. For RACE, the judge model (judge_model) compares the report under test against a reference report criterion by criterion to grade report quality; for FACT, fact_judge_model works with Jina Reader to fetch the cited pages and check whether the citations in the report support their claims. The judge and the model under test are two separate endpoints; judge_model must be specified explicitly.

Upstream ships an evaluator only and places no constraints on the inference side (no limits on tools, turns, or length); its leaderboard entries come from the real outputs of various deep-research products. Scores from this benchmark are therefore only comparable across runs that share the same harness, the same harness settings, and the same judge model — record all three alongside any number you cite.

The appended citation-format requirement

FACT can only verify citations that a report actually writes out, and an agent given only a query usually produces a report with no URLs anywhere — such a report scores zero on FACT, which says nothing about its real citation ability. So when require_citations is at its default of true, a citation-format requirement is appended after the query (one Chinese version and one English version, selected by the task’s language):

Deliver a complete research report in Markdown.
- Cite every non-obvious claim inline as [source title](https://source-url), placed immediately
  after the sentence it supports, using the real URL you retrieved it from.
- End the report with a numbered list of the sources you cited.

This requirement is appended only to the prompt sent to the model under test; the RACE judge always reads the bare query, so instruction_following is graded against the task’s own requirements rather than the one appended here. [title](url) is also one of the four citation forms natively supported by upstream’s extractor, not a format introduced by this integration. Set require_citations: false to fall back to upstream behaviour and send the bare query only.

RACE: reference-relative scoring

RACE gives no absolute score. Each task ships with a reference report written by a strong deep-research product, plus a weighted criteria tree. Scoring proceeds in two steps:

Cleaning. Citation markers, reference lists, and footnotes are first removed from the report under test so the judge compares prose rather than bibliographies. Reports too long for a single call are split at paragraph boundaries and cleaned in parallel. The reference reports ship pre-cleaned and need no reprocessing. Set skip_cleaning: true to skip this step and grade the raw report.
Judging. Within a single call, the judge scores both reports 0-10 against each criterion. Per-criterion scores are first folded into four dimension scores using the criterion weights, then composed into a task total using the dimension weights.

The number finally reported is the ratio target / (target + reference): 0.5 means it tied the reference report, above 0.5 means it beat the reference report, and below 0.5 means it lost to the reference report. The four dimensions — comprehensiveness, insight, instruction_following, and readability — are reported as the same ratio. Required Judge request or schema failures produce FATAL and use the shared evaluation retry budget. A final FATAL invalidates the task and suppresses official run scores; valid tasks may contribute to an explicitly labeled reference score.

FACT: citation grounding

FACT checks whether every citation in the report truly supports the claim it accompanies. All four stages run against the raw report, citation markers intact:

Extract. (fact, ref_idx, url) triples are pulled out of the report body; all four citation forms — [title](url), [15], text 15, and [15†L10] — are recognised.
Deduplicate. Triples are grouped by URL; near-identical statements within a group are collapsed into one.
Scrape. Each unique URL is fetched through Jina Reader. Fetched pages are cached under the AgentCompass data root and reused across runs (scrape_cache).
Validate. Each statement is labelled supported, unsupported, or unknown against the fetched page.

Two exclusion rules match upstream: statements labelled unknown (dead link, paywall, page not found) leave both the numerator and the denominator; a report from which no citation could be extracted is dropped from the FACT averages entirely rather than scored zero.

Parameters

Pass a JSON object via --benchmark-params '{...}', or configure the fields under benchmarks.deepresearch_bench in a YAML file given to --config; explicit CLI values win on shared keys. See the Benchmark overview for merge precedence.

Parameter reference

Parameter	Type	Default	Choices / values	Description
`judge_model`	dict	`null`	`id`, `base_url`, `api_key`, `api_protocol`, `params`	Judge model spec, required (see Judge model spec). It decides RACE grading, and is not the CLI `—model-*`; it also serves as the default model for the cleaning and FACT stages.
`metrics`	list	`[“race”, “fact”]`	`race`, `fact`, or both	Which scoring frameworks to run. Both by default, matching upstream’s `run_benchmark.sh`; set it to `[“race”]` when you only want report quality.
`jina_api_key`	string	`$JINA_API_KEY`	Jina Reader key	Used by FACT to fetch cited pages. Required unless `metrics` is `[“race”]`; a missing key fails at config-building time.
`fact_judge_model`	dict	`null`	same as `judge_model`	Judge for the FACT stages; falls back to `judge_model` when unset.
`cleaning_model`	dict	`null`	same as `judge_model`	Model that performs cleaning before judging; falls back to `judge_model` when unset.
`language`	string	`”all”`	`all` / `zh` / `en`	Filter tasks by query language; `all` = no filter. 50 tasks each in Chinese and English.
`category`	string / list	`”all"`	`"all”`, a single topic name, or a list of topic names (22 listed below)	Filter tasks by topic; `“all”` = no filter. A list takes the union.
`limit`	int	`0`	`0` = no limit	Run only the first N tasks after language and category filtering. Prefer `sample_ids` for a stable smoke-test selection.
`data_dir`	string	`""`	local repository or `data/` directory	Use an existing DeepResearch Bench checkout instead of downloading the default archive.
`dataset_zip_url`	string	official repository archive	ZIP URL	Advanced dataset-source override used only when `data_dir` is empty.
`require_citations`	bool	`true`	`true` / `false`	Whether to append the citation-format requirement after the query (see The appended citation-format requirement). With `false`, only the bare query is sent and FACT usually has nothing to verify.
`skip_cleaning`	bool	`false`	`true` / `false`	Skip cleaning and grade the raw report. Saves one LLM call per task, but shifts the scores.
`pass_threshold`	float	`0.5`	`0.0`-`1.0`	Minimum active primary score required for `passed=true`. With RACE enabled, the default means “tied or beat the reference report”; in a FACT-only run, it means citation accuracy of at least 50%.
`max_retries`	int	`10`	`≥ 1`	Retry budget for one RACE judge call, covering both unparsable JSON and missing dimensions.
`scrape_cache`	bool	`true`	`true` / `false`	Whether to cache fetched pages under the data root and reuse them across runs.
`max_urls`	int	`0`	`0` = no limit	Cap on unique URLs verified per task. A non-zero value bounds cost but drops some citations; the drop is logged.
`max_url_content_chars`	int	`0`	`0` = no truncation	Truncate each fetched page to this length before validation.
`clean_concurrency`	int	`4`	`≥ 1`	Concurrent cleaning calls within one task; only takes effect when a long report is chunked. Cross-task concurrency is controlled by `—task-concurrency`.
`scrape_concurrency`	int	`4`	`≥ 1`	Concurrent Jina Reader fetches within one task.
`fact_llm_concurrency`	int	`4`	`≥ 1`	Concurrent FACT judge calls within one task, covering the extract, deduplicate, and validate stages.

Shared Benchmark fields such as sample_ids follow Benchmark Parameters. DeepResearchBench declares scalar primary metric score, auxiliary binary metric passed, and RACE dimensions and citation statistics as auxiliary scalar observations. At k>1, use the avg execution strategy; selecting pass for a scalar primary fails preflight. See Metrics and Aggregation.

All 22 category values (click to expand)

Science & Technology (16), Finance & Business (14), Software Development (10), Education & Jobs (8), Health (8), Literature (4), History (4), Hardware (4), Industrial (4), Art & Design (4), Games (2), Crime & Law (2), Entertainment (2), Sports & Fitness (2), Software (2), Transportation (2), Religion (2), Home & Hobbies (2), Travel (2), Food & Dining (2), Fashion & Beauty (2), Social Life (2). Numbers in parentheses are the task count per topic (100 total, half Chinese and half English). Case and spacing must match exactly.

Judge model spec

judge_model is passed as a dict with the fields id, base_url, api_key, api_protocol, and params, pointing to the judge model’s own endpoint, with inference parameters under params. We recommend fixing a single judge ** across all models under test. RACE grading is the single biggest factor on the score, so switching judges makes scores no longer comparable across models; likewise, the model under test should not serve as its own judge, as that is neither fair nor comparable. Unlike benchmarks with relatively objective criteria such as DeepSearchQA, the RACE judge also needs a ** large enough context window: one judging call must hold two complete research reports plus the full criteria list, routinely exceeding 100k tokens; a judge that rejects the request outright will burn the whole retry budget, and the task ends up recorded as an error. Choose one capable long-context judge and keep it fixed across the models under test. Upstream’s leaderboard uses gpt-5.5 for RACE and gpt-5.4-mini for FACT, so numbers produced with a different judge are internally comparable but cannot be aligned directly with that leaderboard.

Run examples

The three positional arguments to agentcompass run are Benchmark, Harness, and Model. These examples use deepresearch_bench, naive_search_agent, and $MODEL_NAME, with host_process as the Environment. Set the following environment variables in your terminal before running the examples:

Model under test: MODEL_NAME, MODEL_BASE_URL, and MODEL_API_KEY; see Model connection details.
Judge Model: JUDGE_MODEL_NAME, JUDGE_MODEL_BASE_URL, and JUDGE_MODEL_API_KEY. Use a fixed, independent judge configuration.
Search tools: SERPER_API_KEY for search and JINA_API_KEY for visit.

See the run command for configuration ownership and CLI overrides. The Harness uses jina_api_key for the visit tool; the Benchmark uses it to fetch cited pages for FACT. Both use JINA_API_KEY below. A RACE-only run still needs this key for the agent to read web pages.

Smoke test (single task end-to-end)
Custom parameters
AgentCompass recommended config

Use sample_ids to evaluate a single task, verifying that the end-to-end inference, RACE, and FACT flow works; defaults for the rest.

agentcompass run \
  deepresearch_bench \
  naive_search_agent \
  "$MODEL_NAME" \
  --env host_process \
  --benchmark-params '{
    "judge_model": {
      "id": "'"$JUDGE_MODEL_NAME"'",
      "base_url": "'"$JUDGE_MODEL_BASE_URL"'",
      "api_key": "'"$JUDGE_MODEL_API_KEY"'"
    },
    "jina_api_key": "${JINA_API_KEY}",
    "sample_ids": ["1"]
  }' \
  --harness-params '{
    "serper_api_key": "${SERPER_API_KEY}",
    "jina_api_key": "${JINA_API_KEY}"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Evaluate only the Chinese tasks and skip FACT, to focus on report quality; FACT needs no Jina configuration when metrics is ["race"], while the Harness still uses JINA_API_KEY for visit.

agentcompass run \
  deepresearch_bench \
  naive_search_agent \
  "$MODEL_NAME" \
  --env host_process \
  --benchmark-params '{
    "metrics": ["race"],
    "judge_model": {
      "id": "'"$JUDGE_MODEL_NAME"'",
      "base_url": "'"$JUDGE_MODEL_BASE_URL"'",
      "api_key": "'"$JUDGE_MODEL_API_KEY"'"
    },
    "language": "zh"
  }' \
  --harness-params '{
    "serper_api_key": "${SERPER_API_KEY}",
    "jina_api_key": "${JINA_API_KEY}"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --task-concurrency 8

Evaluate all Chinese and English tasks with both RACE and FACT. Report tasks search more broadly than QA ones, so max_iterations is widened from the default 50 to 80 and max_tool_response_length raised to 16384 so source URLs in the visit tool’s summaries are not truncated; the whole report has to be written in one generation, and its length cap comes from max_tokens in --model-params. RACE and FACT share judge_model by default.

agentcompass run \
  deepresearch_bench \
  naive_search_agent \
  "$MODEL_NAME" \
  --env host_process \
  --benchmark-params '{
    "judge_model": {
      "id": "'"$JUDGE_MODEL_NAME"'",
      "base_url": "'"$JUDGE_MODEL_BASE_URL"'",
      "api_key": "'"$JUDGE_MODEL_API_KEY"'"
    },
    "jina_api_key": "${JINA_API_KEY}"
  }' \
  --harness-params '{
    "serper_api_key": "${SERPER_API_KEY}",
    "jina_api_key": "${JINA_API_KEY}",
    "max_iterations": 80,
    "max_tool_response_length": 16384
  }' \
  --model-params '{
    "max_tokens": 32768
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --task-concurrency 8

Evaluation Results

For shared result conventions, see Run Directory, Aggregate Scores, and Task Files and Shared Fields.

Scoring Metrics

DeepResearchBench’s primary metric is scalar score, ranging from 0 to 1; higher is better. With RACE enabled, it uses overall_score from the relative scoring process described above. A FACT-only run uses citation accuracy, so changing scoring modes changes what the primary score means.

Metric	Type and meaning
`score`	Scalar primary metric selected by the scoring mode.
`passed`	Binary auxiliary metric: the answer is nonempty and its available primary score reaches `pass_threshold`.
`overall_score`	Scalar RACE total. Rubric and dimension weights apply before relative normalization, so averaging the four displayed dimensions does not reconstruct it.
`comprehensiveness` / `insight` / `instruction_following` / `readability`	The four scalar RACE relative scores, explained in the RACE section above.
`citation_accuracy`	FACT citation accuracy: supported citations / checked citations.
`citations_checked` / `citations_supported` / `citations_total`	Scalar counts of checked, supported, and initially extracted citations.
`fact_scored`	A 0/1 scalar flag for a scored FACT task; omitted when FACT reports an error.

Overall FACT citation_accuracy is weighted by citation count: total supported citations divided by total checked citations. FACT-only runs use the same formula for overall score. Derived metrics avg_citations and avg_effective_citations divide total checked and supported citations, respectively, by the number of scored FACT tasks. RACE and FACT can cover different tasks and need not share a denominator. Tasks with no extracted citations (no_citations_found) do not enter FACT averages. Tasks with extracted citations whose verdicts are all unknown still count as scored tasks, with 0 checked citations. The scalar primary metric does not support the pass execution strategy; auxiliary passed only reports threshold attainment. See Metrics and Aggregation for repeated attempts, category aggregation, and shared scoring failure rules.

Task Results and Scoring Evidence

Within each attempt’s meta.benchmark, scoring stores RACE and FACT evidence, the effective pass_threshold, and the passed verdict when available. Fields below are relative to scoring:

Field	Meaning
`race.overall_score` and the four dimensions	The task’s relative scores
`race.raw`	Pre-normalization weighted sums for both reports, plus any criterion matched only via fuzzy matching
`race.cleaning`	`applied` or `skipped`
`race.article_chars` / `race.cleaned_article_chars`	Report length before and after cleaning, useful for checking whether cleaning shortened the report unexpectedly
`race.judge_attempts`	How many judge calls this task actually consumed
`race.error`	RACE scoring failure, such as `cleaning_failed`, `judge_failed`, or `scoring_failed`
`race.reason` / `fact.reason`	`empty_answer` for an empty answer; the corresponding scorer does not call the judge
`fact.n_citations` / `fact.unique_urls`	Extracted citation triples and deduplicated URLs selected for verification; a positive `max_urls` caps the latter
`fact.citations_checked` / `citations_supported`	Statements that reached a verdict, and how many of them were supported
`fact.scrape_failures` / `validate_failures`	Unreachable pages, and validation calls that never returned usable JSON
`fact.citations`	Per-URL statements, verdicts, and errors, for tracing a score back to a specific page
`fact.error`	Why FACT has no valid result, such as `no_citations_found`, `extraction_failed`, or a page scraping or citation validation error

Using AgentCompass

Modules

Other Features

How it works

Inference and scoring

The appended citation-format requirement

RACE: reference-relative scoring

FACT: citation grounding

Parameters

Parameter reference

Judge model spec

Run examples

Evaluation Results

Scoring Metrics

Task Results and Scoring Evidence

​How it works

​Inference and scoring

​The appended citation-format requirement

​RACE: reference-relative scoring

​FACT: citation grounding

​Parameters

​Parameter reference

​Judge model spec

​Run examples

​Evaluation Results

​Scoring Metrics

​Task Results and Scoring Evidence

How it works

Inference and scoring

The appended citation-format requirement

RACE: reference-relative scoring

FACT: citation grounding

Parameters

Parameter reference

Judge model spec

Run examples

Evaluation Results

Scoring Metrics

Task Results and Scoring Evidence