{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-03903","title":"CC-OCR V2: Fine-Grained Attribution of LMM Failures in Real-World Visual Document Understanding","arxiv_id":"2605.03903","date":"2026-05-05","proceeding":null,"authors":["Chunyi Peng","Zhipeng Xu","Yuqi Xiong","Zulong Chen","Junhao Ji","Qing Liu","Zhenghao Liu","Zubao Qin","Bing Zhao","Jianqiang Wan","Jun Tang","Zhibo Yang","Shuai Bai","Dayiheng Liu","Maosong Sun"],"abstract":"Recent Large Multimodal Models (LMMs) have achieved remarkable progress on OCR-centric document understanding and processing tasks. Existing benchmarks primarily evaluate LMMs across diverse tasks to reflect practical document-processing workflows or analyze how document characteristics influence model performance. However, they provide limited insight into the reliability of LMMs under real-world document acquisition conditions, where factors such as lighting, screen displays, imaging quality, and capture methods can substantially affect performance. To bridge this gap, we present CC-OCR v2, a comprehensive benchmark for attributing LMM failures in real-world document processing. CC-OCR v2 provides a unified evaluation framework covering five core document-processing capabilities through 16 subtasks, 74 application scenarios, and 7,093 samples. The benchmark spans five evaluation tracks, ten document categories, and 32 languages in the recognition suite. Beyond task-level evaluation, each sample is annotated with ten fine-grained document factors, enabling systematic attribution of model failures across document types and acquisition conditions. Extensive experiments on 17 representative LMMs reveal substantial performance variation across tasks, document categories, and real-world conditions. Moreover, models with comparable overall accuracy often exhibit fundamentally different failure patterns under specific document factors, highlighting a significant gap between benchmark-level performance and reliable deployment in practical applications. The dataset and evaluation toolkit are publicly available at https://github.com/eioss/CC-OCR-V2.","url_abs":"https://arxiv.org/abs/2605.03903","url_pdf":"https://arxiv.org/pdf/2605.03903","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.03903"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/eioss/CC-OCR-V2","reach":null}],"summary":{"ran":15,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":20,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d18aa3036d3fc5e9","entry":"download_file","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_funsd.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_funsd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d18aa3036d3fc5e9"}},{"code_sha256_prefix":"670ab7d42b401f10","entry":"download_gdrive_file","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_poie.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_poie.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"670ab7d42b401f10"}},{"code_sha256_prefix":"98691b6e1f5015c0","entry":"extract_archive","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_cell.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_cell.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98691b6e1f5015c0"}},{"code_sha256_prefix":"4890e4c321389b25","entry":"extract_image_filenames","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_cell.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_cell.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4890e4c321389b25"}},{"code_sha256_prefix":"a13983023805e352","entry":"extract_images_from_parquet","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_nanonets_kie.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_nanonets_kie.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a13983023805e352"}},{"code_sha256_prefix":"e7964453b0bde023","entry":"extract_images_zip","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_sibr.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_sibr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7964453b0bde023"}},{"code_sha256_prefix":"0ba5dc84418ccf8c","entry":"find_and_extract_archives","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_funsd.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_funsd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ba5dc84418ccf8c"}},{"code_sha256_prefix":"350f7dc40d87cb88","entry":"find_and_extract_archives","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_poie.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_poie.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"350f7dc40d87cb88"}},{"code_sha256_prefix":"1fca150d4fb436ea","entry":"find_images_zip","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_sibr.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_sibr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1fca150d4fb436ea"}},{"code_sha256_prefix":"f3d2ead79164c378","entry":"find_parquet_files","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_nanonets_kie.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_nanonets_kie.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3d2ead79164c378"}},{"code_sha256_prefix":"3f558876e0aa525a","entry":"find_pdf_file","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_deepform.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_deepform.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f558876e0aa525a"}},{"code_sha256_prefix":"2d9c80fa434a258e","entry":"find_pdf_file","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_docile.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_docile.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d9c80fa434a258e"}},{"code_sha256_prefix":"ba47656355c304bb","entry":"load_label_json","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_cell.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_cell.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba47656355c304bb"}},{"code_sha256_prefix":"7d7bc58a6b0eecac","entry":"pdf_to_images","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_docile.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_docile.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d7bc58a6b0eecac"}},{"code_sha256_prefix":"5a2559ab82d74b9c","entry":"run_command","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_sibr.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_sibr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a2559ab82d74b9c"}},{"code_sha256_prefix":"30065dd0ad9b7ba0","entry":"download_huggingface_dataset","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_nanonets_kie.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_nanonets_kie.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30065dd0ad9b7ba0"}},{"code_sha256_prefix":"a5112d4e98b61cbc","entry":"download_huggingface_dataset","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_cord.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_cord.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5112d4e98b61cbc"}},{"code_sha256_prefix":"07b4febfa68aee51","entry":"download_huggingface_tsv","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_ephoie.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_ephoie.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07b4febfa68aee51"}},{"code_sha256_prefix":"cfe0050f34d6c16d","entry":"extract_images_from_parquet","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_cord.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_cord.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfe0050f34d6c16d"}},{"code_sha256_prefix":"ab8b09f42c6e3905","entry":"extract_images_from_tsv","repo":"eioss/CC-OCR-V2","repo_kind":"found_in_text","path":"datasets_process/process_ephoie.py","file_url":"https://github.com/eioss/CC-OCR-V2/blob/HEAD/datasets_process/process_ephoie.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab8b09f42c6e3905"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}