{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-08888","title":"DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding","arxiv_id":"2605.08888","date":"2026-05-09","proceeding":null,"authors":["Xiang Feng","Jiawei Zhou","Zhangfeng Huang","Kewei Wang","Shanshan Ye","Jinxin Hu","Zulong Chen","Yong Luo","Jing Zhang"],"abstract":"Evaluating whether Multimodal Large Language Models can produce trustworthy, verifiable reasoning over long, visually rich documents requires evaluation beyond end-to-end answer accuracy. We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem: given a complete PDF document and a question, the model outputs evidence pages, supporting evidence regions, relevant factual statements, and a final answer. We design a four-stage evaluation protocol -- Page Localization, Region Grounding, Fact Extraction, and Answer Verification -- that audits each level of the trajectory independently through inter-stage decoupling, with all judges selected and calibrated via human alignment studies. DocScope comprises 1,124 questions derived from 273 documents, with all hierarchical evidence annotations completed by human annotators. We benchmark 6 proprietary models, 12 open-weight models, and several domain-specific systems. Our experiments reveal that answer accuracy cannot substitute for trajectory-level evaluation: even among correct answers, the highest observed rate of complete evidence chains is only 29\\%. Across all models, region grounding remains the weakest trajectory stage. Furthermore, the primary difficulty stems from aggregating evidence dispersed across long distances and multiple document clusters, while an oracle study identifies faithful perception and fact extraction as the dominant capability bottleneck. Cross-architecture comparisons further suggest that activated parameter count matters more than total scale. The benchmark and code will be publicly released at https://github.com/MiliLab/DocScope.","url_abs":"https://arxiv.org/abs/2605.08888","url_pdf":"https://arxiv.org/pdf/2605.08888","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.08888","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.08888"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/MiliLab/DocScope","reach":null}],"summary":{"ran":2,"ran_draft_wrong":6,"ran_fixture":1,"ran_honours":1,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":14,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"349c03947130361a","entry":"Citation","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"349c03947130361a"}},{"code_sha256_prefix":"422a0227ede97d97","entry":"_Mask","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"422a0227ede97d97"}},{"code_sha256_prefix":"d2f7410083b41ac7","entry":"_apply_mask","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2f7410083b41ac7"}},{"code_sha256_prefix":"884005fcb5a1f515","entry":"_attribute_and_render","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"884005fcb5a1f515"}},{"code_sha256_prefix":"f5f52d8b4b5db9e1","entry":"_bbox_is_normalized","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f5f52d8b4b5db9e1"}},{"code_sha256_prefix":"4ae2acba664e409d","entry":"_clean_sentence","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ae2acba664e409d"}},{"code_sha256_prefix":"3b61053b6054a8f6","entry":"_parse_citation_match","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b61053b6054a8f6"}},{"code_sha256_prefix":"a8ab925a3eb0bd33","entry":"_protect_abbreviations","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8ab925a3eb0bd33"}},{"code_sha256_prefix":"b7c8cc61ff19b313","entry":"_regex_split_sentences","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7c8cc61ff19b313"}},{"code_sha256_prefix":"d85bd4d72ee4e3a7","entry":"_restore_non_citation","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d85bd4d72ee4e3a7"}},{"code_sha256_prefix":"1231d04011a65748","entry":"_icu_split_sentences","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1231d04011a65748"}},{"code_sha256_prefix":"2da663d6e555af68","entry":"extract_citations","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2da663d6e555af68"}},{"code_sha256_prefix":"7f384f70c7f69c5a","entry":"mask_regions","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f384f70c7f69c5a"}},{"code_sha256_prefix":"c33b421f2b37e992","entry":"split_sentences","repo":"MiliLab/DocScope","repo_kind":"found_in_text","path":"code/infer/reasoning_prompts.py","file_url":"https://github.com/MiliLab/DocScope/blob/HEAD/code/infer/reasoning_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c33b421f2b37e992"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}