{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-cot-unleashing-chain-of-thought","title":"Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning","arxiv_id":"2403.16999","date":"2024-03-25","proceeding":null,"authors":["Hao Shao","Shengju Qian","Han Xiao","Guanglu Song","Zhuofan Zong","Letian Wang","Yu Liu","Hongsheng Li"],"abstract":"Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. However, they often lack interpretability and struggle with complex visual inputs, especially when the resolution of the input image is high or when the interested region that could provide key information for answering the question is small. To address these challenges, we collect and introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions. Additionally, about 98k pairs of them are annotated with detailed reasoning steps. Importantly, we propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts. We also introduce the related benchmark to evaluate the MLLMs in scenarios requiring specific local region identification. Extensive experiments demonstrate the effectiveness of our framework and shed light on better inference strategies. The Visual CoT dataset, benchmark, and pre-trained models are available on https://hao-shao.com/projects/viscot.html to support further research in this area.","url_abs":"https://arxiv.org/abs/2403.16999v3","url_pdf":"https://arxiv.org/pdf/2403.16999v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visual-cot-unleashing-chain-of-thought","repo_url":"https://github.com/deepcs233/visual-cot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.16999","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.16999"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepcs233/visual-cot","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"ran":4,"ran_violates":2},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"deepcs233/visual-cot","repo_kind":"official","path":"llava/eval/model_cot_det_loader.py","file_url":"https://github.com/deepcs233/visual-cot/blob/HEAD/llava/eval/model_cot_det_loader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"fb48214e8ea84c17","entry":"get_eval","repo":"deepcs233/visual-cot","repo_kind":"official","path":"llava/eval/eval_gpt_review_visual.py","file_url":"https://github.com/deepcs233/visual-cot/blob/HEAD/llava/eval/eval_gpt_review_visual.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb48214e8ea84c17"}},{"code_sha256_prefix":"5b700f36a74a7df5","entry":"get_score","repo":"deepcs233/visual-cot","repo_kind":"official","path":"llava/eval/eval_cot_score.py","file_url":"https://github.com/deepcs233/visual-cot/blob/HEAD/llava/eval/eval_cot_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5b700f36a74a7df5"}},{"code_sha256_prefix":"bae18947b56f2be1","entry":"is_none","repo":"deepcs233/visual-cot","repo_kind":"official","path":"llava/eval/model_vqa_mmbench.py","file_url":"https://github.com/deepcs233/visual-cot/blob/HEAD/llava/eval/model_vqa_mmbench.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bae18947b56f2be1"}},{"code_sha256_prefix":"bbe3f02e93e14ae0","entry":"load_image","repo":"deepcs233/visual-cot","repo_kind":"official","path":"llava/eval/model_vqa_qbench.py","file_url":"https://github.com/deepcs233/visual-cot/blob/HEAD/llava/eval/model_vqa_qbench.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bbe3f02e93e14ae0"}},{"code_sha256_prefix":"702fe5e07e3728bd","entry":"make_request_openai","repo":"deepcs233/visual-cot","repo_kind":"official","path":"llava/eval/eval_cot_score.py","file_url":"https://github.com/deepcs233/visual-cot/blob/HEAD/llava/eval/eval_cot_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"702fe5e07e3728bd"}},{"code_sha256_prefix":"763f6fe20bd52fbb","entry":"parse_score","repo":"deepcs233/visual-cot","repo_kind":"official","path":"llava/eval/eval_gpt_review_visual.py","file_url":"https://github.com/deepcs233/visual-cot/blob/HEAD/llava/eval/eval_gpt_review_visual.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"763f6fe20bd52fbb"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"deepcs233/visual-cot","repo_kind":"official","path":"llava/eval/model_cot_det_loader.py","file_url":"https://github.com/deepcs233/visual-cot/blob/HEAD/llava/eval/model_cot_det_loader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}