{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mc-cot-a-modular-collaborative-cot-framework","title":"MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integration","arxiv_id":"2410.04521","date":"2024-10-06","proceeding":null,"authors":["Lai Wei","Wenkai Wang","Xiaoyu Shen","Yu Xie","Zhihao Fan","Xiaojin Zhang","Zhongyu Wei","Wei Chen"],"abstract":"In recent advancements, multimodal large language models (MLLMs) have been fine-tuned on specific medical image datasets to address medical visual question answering (Med-VQA) tasks. However, this common approach of task-specific fine-tuning is costly and necessitates separate models for each downstream task, limiting the exploration of zero-shot capabilities. In this paper, we introduce MC-CoT, a modular cross-modal collaboration Chain-of-Thought (CoT) framework designed to enhance the zero-shot performance of MLLMs in Med-VQA by leveraging large language models (LLMs). MC-CoT improves reasoning and information extraction by integrating medical knowledge and task-specific guidance, where LLM provides various complex medical reasoning chains and MLLM provides various observations of medical images based on instructions of the LLM. Our experiments on datasets such as SLAKE, VQA-RAD, and PATH-VQA show that MC-CoT surpasses standalone MLLMs and various multimodality CoT frameworks in recall rate and accuracy. These findings highlight the importance of incorporating background information and detailed guidance in addressing complex zero-shot Med-VQA tasks.","url_abs":"https://arxiv.org/abs/2410.04521v1","url_pdf":"https://arxiv.org/pdf/2410.04521v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mc-cot-a-modular-collaborative-cot-framework","repo_url":"https://github.com/thomaswei-cn/MC-CoT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"medical-visual-question-answering","task_name":"Medical Visual Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.04521","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.04521"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thomaswei-cn/MC-CoT","reach":{"status":"ok"}}],"summary":{"ran":15},"by_repo_kind":{"official":{"samples":15,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"6b92bbd5846da32b","entry":"calculate","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"evaluation/eval_recall.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/evaluation/eval_recall.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b92bbd5846da32b"}},{"code_sha256_prefix":"90496e55127efecd","entry":"get_description_prompt","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/MCCoT.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/MCCoT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90496e55127efecd"}},{"code_sha256_prefix":"64df14f409178414","entry":"get_final_prompt","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/IICoT.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/IICoT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64df14f409178414"}},{"code_sha256_prefix":"5175e8cc0f49f1c7","entry":"get_llm_guide","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/IICoT.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/IICoT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5175e8cc0f49f1c7"}},{"code_sha256_prefix":"9c7f5d246b874c50","entry":"get_prompt","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/VisualOnly.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/VisualOnly.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9c7f5d246b874c50"}},{"code_sha256_prefix":"000be3865cec37b5","entry":"get_prompt_1","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/DDCoT.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/DDCoT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"000be3865cec37b5"}},{"code_sha256_prefix":"0af7dfa170b8908b","entry":"get_prompt_1","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/MMCoT.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/MMCoT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0af7dfa170b8908b"}},{"code_sha256_prefix":"27f9b05e71702fe8","entry":"get_prompt_2","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/DDCoT.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/DDCoT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27f9b05e71702fe8"}},{"code_sha256_prefix":"73e8a59dd51c7a69","entry":"get_prompt_2","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/MMCoT.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/MMCoT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73e8a59dd51c7a69"}},{"code_sha256_prefix":"196a35d098174fae","entry":"get_qvix_prompt_stg1","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/Qvix.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/Qvix.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"196a35d098174fae"}},{"code_sha256_prefix":"0257d60a33d66847","entry":"get_qvix_prompt_stg2","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/Qvix.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/Qvix.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0257d60a33d66847"}},{"code_sha256_prefix":"cf691b4b36db3ef4","entry":"get_rationale_with_guide","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/IICoT.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/IICoT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cf691b4b36db3ef4"}},{"code_sha256_prefix":"66b70633eccfaedb","entry":"get_total_score_with_len","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"evaluation/eval_show.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/evaluation/eval_show.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66b70633eccfaedb"}},{"code_sha256_prefix":"0fbe28a9f28a96ab","entry":"is_module_required","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/MCCoT.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/MCCoT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fbe28a9f28a96ab"}},{"code_sha256_prefix":"504831ad705b7b9d","entry":"match_tasks","repo":"thomaswei-cn/MC-CoT","repo_kind":"official","path":"method/MCCoT.py","file_url":"https://github.com/thomaswei-cn/MC-CoT/blob/HEAD/method/MCCoT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"504831ad705b7b9d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}