{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/enhancing-human-like-multi-modal-reasoning-a","title":"Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework","arxiv_id":"2307.12626","date":"2023-07-24","proceeding":null,"authors":["Jingxuan Wei","Cheng Tan","Zhangyang Gao","Linzhuang Sun","Siyuan Li","Bihui Yu","Ruifeng Guo","Stan Z. Li"],"abstract":"Multimodal reasoning is a critical component in the pursuit of artificial intelligence systems that exhibit human-like intelligence, especially when tackling complex tasks. While the chain-of-thought (CoT) technique has gained considerable attention, the existing ScienceQA dataset, which focuses on multimodal scientific questions and explanations from elementary and high school textbooks, lacks a comprehensive evaluation of diverse approaches. To address this gap, we present COCO Multi-Modal Reasoning(COCO-MMR) dataset, a novel dataset that encompasses an extensive collection of open-ended questions, rationales, and answers derived from the large object dataset COCO. Unlike previous datasets that rely on multiple-choice questions, our dataset pioneers the use of open-ended questions in the context of multimodal CoT, introducing a more challenging problem that effectively assesses the reasoning capability of CoT models. Through comprehensive evaluations and detailed analyses, we provide valuable insights and propose innovative techniques, including multi-hop cross-modal attention and sentence-level contrastive learning, to enhance the image and text encoders. Extensive experiments demonstrate the efficacy of the proposed dataset and techniques, offering novel perspectives for advancing multimodal reasoning. The data and code are available at \\href{https://github.com/weijingxuan/COCO-MMR}{https://github.com/weijingxuan/COCO-MMR}.","url_abs":"https://arxiv.org/abs/2307.12626v2","url_pdf":"https://arxiv.org/pdf/2307.12626v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"enhancing-human-like-multi-modal-reasoning-a","repo_url":"https://github.com/weijingxuan/COCO-MMR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"multimodal-reasoning","task_name":"Multimodal Reasoning"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.12626","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.12626"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/weijingxuan/COCO-MMR","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_draft_wrong":4,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"ab795c060d39c540","entry":"bleu_score","repo":"weijingxuan/COCO-MMR","repo_kind":"official","path":"evaluations.py","file_url":"https://github.com/weijingxuan/COCO-MMR/blob/HEAD/evaluations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab795c060d39c540"}},{"code_sha256_prefix":"07349e17621c2b32","entry":"caculate_bleu","repo":"weijingxuan/COCO-MMR","repo_kind":"official","path":"evaluations.py","file_url":"https://github.com/weijingxuan/COCO-MMR/blob/HEAD/evaluations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07349e17621c2b32"}},{"code_sha256_prefix":"7a8dffc83d89cedd","entry":"get_choice_text","repo":"weijingxuan/COCO-MMR","repo_kind":"official","path":"utils_prompt.py","file_url":"https://github.com/weijingxuan/COCO-MMR/blob/HEAD/utils_prompt.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a8dffc83d89cedd"}},{"code_sha256_prefix":"85f084f6054a421e","entry":"get_context_text","repo":"weijingxuan/COCO-MMR","repo_kind":"official","path":"utils_prompt.py","file_url":"https://github.com/weijingxuan/COCO-MMR/blob/HEAD/utils_prompt.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85f084f6054a421e"}},{"code_sha256_prefix":"c86a79e229253816","entry":"get_question_text","repo":"weijingxuan/COCO-MMR","repo_kind":"official","path":"utils_prompt.py","file_url":"https://github.com/weijingxuan/COCO-MMR/blob/HEAD/utils_prompt.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c86a79e229253816"}},{"code_sha256_prefix":"c191534b49494742","entry":"load_image_features","repo":"weijingxuan/COCO-MMR","repo_kind":"official","path":"utils_data.py","file_url":"https://github.com/weijingxuan/COCO-MMR/blob/HEAD/utils_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c191534b49494742"}},{"code_sha256_prefix":"e605721addd552ac","entry":"tokenize","repo":"weijingxuan/COCO-MMR","repo_kind":"official","path":"evaluations.py","file_url":"https://github.com/weijingxuan/COCO-MMR/blob/HEAD/evaluations.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e605721addd552ac"}},{"code_sha256_prefix":"c16954948a348310","entry":"extract_features_batch","repo":"weijingxuan/COCO-MMR","repo_kind":"official","path":"image_progress_coco.py","file_url":"https://github.com/weijingxuan/COCO-MMR/blob/HEAD/image_progress_coco.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c16954948a348310"}},{"code_sha256_prefix":"6122688d8c359cc9","entry":"init_mydata","repo":"weijingxuan/COCO-MMR","repo_kind":"official","path":"image_progress_coco.py","file_url":"https://github.com/weijingxuan/COCO-MMR/blob/HEAD/image_progress_coco.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6122688d8c359cc9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}