{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deepresearch-bench-a-comprehensive-benchmark","title":"DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents","arxiv_id":"2506.11763","date":"2025-06-13","proceeding":null,"authors":["Mingxuan Du","Benfeng Xu","Chiwei Zhu","Xiaorui Wang","Zhendong Mao"],"abstract":"Deep Research Agents are a prominent category of LLM-based agents. By autonomously orchestrating multistep web exploration, targeted retrieval, and higher-order synthesis, they transform vast amounts of online information into analyst-grade, citation-rich reports--compressing hours of manual desk research into minutes. However, a comprehensive benchmark for systematically evaluating the capabilities of these agents remains absent. To bridge this gap, we present DeepResearch Bench, a benchmark consisting of 100 PhD-level research tasks, each meticulously crafted by domain experts across 22 distinct fields. Evaluating DRAs is inherently complex and labor-intensive. We therefore propose two novel methodologies that achieve strong alignment with human judgment. The first is a reference-based method with adaptive criteria to assess the quality of generated research reports. The other framework is introduced to evaluate DRA's information retrieval and collection capabilities by assessing its effective citation count and overall citation accuracy. We have open-sourced DeepResearch Bench and key components of these frameworks at https://github.com/Ayanami0730/deep_research_bench to accelerate the development of practical LLM-based agents.","url_abs":"https://arxiv.org/abs/2506.11763v1","url_pdf":"https://arxiv.org/pdf/2506.11763v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deepresearch-bench-a-comprehensive-benchmark","repo_url":"https://github.com/ayanami0730/deep_research_bench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.11763","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.11763"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ayanami0730/deep_research_bench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":10},"by_repo_kind":{"official":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"aa9b182356c68449","entry":"calculate_weighted_scores","repo":"ayanami0730/deep_research_bench","repo_kind":"official","path":"utils/score_calculator.py","file_url":"https://github.com/ayanami0730/deep_research_bench/blob/HEAD/utils/score_calculator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa9b182356c68449"}},{"code_sha256_prefix":"8c4b44ca751e88c7","entry":"call_model","repo":"ayanami0730/deep_research_bench","repo_kind":"official","path":"utils/api.py","file_url":"https://github.com/ayanami0730/deep_research_bench/blob/HEAD/utils/api.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c4b44ca751e88c7"}},{"code_sha256_prefix":"4194b5e6692475d5","entry":"clean_escape","repo":"ayanami0730/deep_research_bench","repo_kind":"official","path":"utils/extract.py","file_url":"https://github.com/ayanami0730/deep_research_bench/blob/HEAD/utils/extract.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4194b5e6692475d5"}},{"code_sha256_prefix":"295db9ae8071476d","entry":"clean_urls","repo":"ayanami0730/deep_research_bench","repo_kind":"official","path":"utils/extract.py","file_url":"https://github.com/ayanami0730/deep_research_bench/blob/HEAD/utils/extract.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"295db9ae8071476d"}},{"code_sha256_prefix":"0c67d85c3cf30824","entry":"format_criteria_list","repo":"ayanami0730/deep_research_bench","repo_kind":"official","path":"deepresearch_bench_race.py","file_url":"https://github.com/ayanami0730/deep_research_bench/blob/HEAD/deepresearch_bench_race.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c67d85c3cf30824"}},{"code_sha256_prefix":"df3522d3429d022b","entry":"parse_llm_output_as_json","repo":"ayanami0730/deep_research_bench","repo_kind":"official","path":"utils/generate_criteria.py","file_url":"https://github.com/ayanami0730/deep_research_bench/blob/HEAD/utils/generate_criteria.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df3522d3429d022b"}},{"code_sha256_prefix":"6f329652e5b07a90","entry":"remove_urls","repo":"ayanami0730/deep_research_bench","repo_kind":"official","path":"utils/extract.py","file_url":"https://github.com/ayanami0730/deep_research_bench/blob/HEAD/utils/extract.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f329652e5b07a90"}},{"code_sha256_prefix":"1617fedf2ce095c2","entry":"round_weights_and_adjust","repo":"ayanami0730/deep_research_bench","repo_kind":"official","path":"utils/generate_criteria.py","file_url":"https://github.com/ayanami0730/deep_research_bench/blob/HEAD/utils/generate_criteria.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1617fedf2ce095c2"}},{"code_sha256_prefix":"5150c824e8fa6aab","entry":"scrape_url","repo":"ayanami0730/deep_research_bench","repo_kind":"official","path":"utils/api.py","file_url":"https://github.com/ayanami0730/deep_research_bench/blob/HEAD/utils/api.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5150c824e8fa6aab"}},{"code_sha256_prefix":"7064953dcffd3450","entry":"validate_weights","repo":"ayanami0730/deep_research_bench","repo_kind":"official","path":"utils/generate_criteria.py","file_url":"https://github.com/ayanami0730/deep_research_bench/blob/HEAD/utils/generate_criteria.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7064953dcffd3450"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}