{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-01358","title":"HopRefusalBench: Diagnosing Refusal Failures in Search-Augmented Agents for Multi-Hop Reasoning","arxiv_id":"2608.01358","date":"2026-08-02","proceeding":null,"authors":["Jianan Xie","Xin Sun","Zhongqi Chen","Xing Zheng","Qiang Liu","Bowen Song"],"abstract":"Search-augmented large language model agents are increasingly capable of solving knowledge-intensive tasks, but their behavior when a multi-hop question is fundamentally unanswerable remains poorly understood. Existing abstention benchmarks largely expose defects at the surface of single-hop queries and therefore cannot reveal failures that emerge only after valid intermediate reasoning and retrieval. We introduce HopRefusalBench, the first controlled benchmark of refusal within multi-hop search, comprising 889 unanswerable questions constructed from KILT-grounded entity paths. It crosses three causes of unanswerability (answer unknown, false premise, and underspecified context) with root, middle, and terminal topologies, making premise verification, intermediate-bridge validation, and terminal stopping separately observable. We further propose a final-outcome taxonomy spanning target-aware refusal, pseudo-refusal, hallucinated completion, and search-budget exhaustion, together with source-aware trajectory metrics for post-trigger continuation and token waste. Across ten frontier proprietary and open-weight models in search-augmented mode, the best model achieves a target-aware correct halting rate (TCHR) of only 42.9%. Root and middle items are consistently harder than terminal items, and all models attain their highest TCHR on false premises and their lowest on underspecified questions. Yet when pooled across categories, 84.7--98.4% of each model's explicit refusal-like responses identify the correct rationale, localizing the main bottleneck to committing to an appropriate non-answer; failed trajectories instead diverge into hallucination or search-budget exhaustion. These results establish refusal in multi-hop search as a consequential evaluation problem and provide a foundation for diagnosing and improving the reliability of search-augmented agents.","url_abs":"https://arxiv.org/abs/2608.01358","url_pdf":"https://arxiv.org/pdf/2608.01358","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2608.01358","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.01358"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/JiananXie/HopRefusalBench","reach":null}],"summary":{"ran":15,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":16,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"f09e500a1edc8746","entry":"canonical_json","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"src/artifacts.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/src/artifacts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f09e500a1edc8746"}},{"code_sha256_prefix":"ec07799bcdb92981","entry":"content_hash","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"src/artifacts.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/src/artifacts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec07799bcdb92981"}},{"code_sha256_prefix":"544339280fed4c2a","entry":"create_quality_checker","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"src/quality_check.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/src/quality_check.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"544339280fed4c2a"}},{"code_sha256_prefix":"b1442ea4af499fa8","entry":"env_float","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"settings.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/settings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1442ea4af499fa8"}},{"code_sha256_prefix":"21b411970a4e55b2","entry":"env_int","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"settings.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/settings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"21b411970a4e55b2"}},{"code_sha256_prefix":"41d4015dedb8727f","entry":"env_str","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"settings.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/settings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"41d4015dedb8727f"}},{"code_sha256_prefix":"5d8985e5778abf46","entry":"extract_responses_text","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"run_inference.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/run_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d8985e5778abf46"}},{"code_sha256_prefix":"5f7fc23d9638482e","entry":"file_sha256","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"src/artifacts.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/src/artifacts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f7fc23d9638482e"}},{"code_sha256_prefix":"90b3f0afb62a2933","entry":"load_dataset_map","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"evaluate.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90b3f0afb62a2933"}},{"code_sha256_prefix":"2820cd4e6747f653","entry":"load_jsonl","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"evaluate.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2820cd4e6747f653"}},{"code_sha256_prefix":"eefa62bb7ec41adf","entry":"normalize_title","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"src/kilt_graph.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/src/kilt_graph.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eefa62bb7ec41adf"}},{"code_sha256_prefix":"3dc37d500670612f","entry":"parse_request_overrides","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"evaluate.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3dc37d500670612f"}},{"code_sha256_prefix":"e5379547957e1ff2","entry":"resolve_output_dir","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"run_experiment.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e5379547957e1ff2"}},{"code_sha256_prefix":"4185b30c8b8be6cd","entry":"safe_name","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"run_experiment.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4185b30c8b8be6cd"}},{"code_sha256_prefix":"0950f92c5c5cf67f","entry":"should_use_responses_api","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"run_inference.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/run_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0950f92c5c5cf67f"}},{"code_sha256_prefix":"b02e7f18870bbece","entry":"load_seeds","repo":"JiananXie/HopRefusalBench","repo_kind":"found_in_text","path":"src/pipeline.py","file_url":"https://github.com/JiananXie/HopRefusalBench/blob/HEAD/src/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b02e7f18870bbece"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}