{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/compositional-chain-of-thought-prompting-for","title":"Compositional Chain-of-Thought Prompting for Large Multimodal Models","arxiv_id":"2311.17076","date":"2023-11-27","proceeding":"CVPR 2024 1","authors":["Chancharik Mitra","Brandon Huang","Trevor Darrell","Roei Herzig"],"abstract":"The combination of strong visual backbones and Large Language Model (LLM) reasoning has led to Large Multimodal Models (LMMs) becoming the current standard for a wide range of vision and language (VL) tasks. However, recent research has shown that even the most advanced LMMs still struggle to capture aspects of compositional visual reasoning, such as attributes and relationships between objects. One solution is to utilize scene graphs (SGs)--a formalization of objects and their relations and attributes that has been extensively used as a bridge between the visual and textual domains. Yet, scene graph data requires scene graph annotations, which are expensive to collect and thus not easily scalable. Moreover, finetuning an LMM based on SG data can lead to catastrophic forgetting of the pretraining objective. To overcome this, inspired by chain-of-thought methods, we propose Compositional Chain-of-Thought (CCoT), a novel zero-shot Chain-of-Thought prompting method that utilizes SG representations in order to extract compositional knowledge from an LMM. Specifically, we first generate an SG using the LMM, and then use that SG in the prompt to produce a response. Through extensive experiments, we find that the proposed CCoT approach not only improves LMM performance on several vision and language VL compositional benchmarks but also improves the performance of several popular LMMs on general multimodal benchmarks, without the need for fine-tuning or annotated ground-truth SGs. Code: https://github.com/chancharikmitra/CCoT","url_abs":"https://arxiv.org/abs/2311.17076v3","url_pdf":"https://arxiv.org/pdf/2311.17076v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"compositional-chain-of-thought-prompting-for","repo_url":"https://github.com/chancharikmitra/ccot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"LLaVA-1.5-CCoT","rank_in_archive_order":31,"of":114,"metrics":{"Group Score":"22.3","Image Score":"35.5","Text Score":"42.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"LLaVA-1.5","rank_in_archive_order":49,"of":114,"metrics":{"Group Score":"20.1","Image Score":"33.3","Text Score":"36.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"LLaVA-1.5-ZS-CoT","rank_in_archive_order":79,"of":114,"metrics":{"Group Score":"12.3","Image Score":"22.5","Text Score":"28.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"InstructBLIP-CCoT","rank_in_archive_order":97,"of":114,"metrics":{"Group Score":"8.3","Image Score":"21.3","Text Score":"21.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"InstructBLIP-ZS-CoT","rank_in_archive_order":113,"of":114,"metrics":{"Group Score":"4.0","Image Score":"16.3","Text Score":"9.3"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"InstructBLIP","rank_in_archive_order":114,"of":114,"metrics":{"Group Score":"3.3","Image Score":"11.5","Text Score":"7.0"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.17076","atlas_url":"https://app.syntology.ai/?focus=2311.17076","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.17076"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chancharikmitra/ccot","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_violates":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"chancharikmitra/ccot","repo_kind":"official","path":"GPT-4V/Sphinx_bench.py","file_url":"https://github.com/chancharikmitra/ccot/blob/HEAD/GPT-4V/Sphinx_bench.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"bae18947b56f2be1","entry":"is_none","repo":"chancharikmitra/ccot","repo_kind":"official","path":"InstructBLIP-13b/InstructBLIP_MMBench.py","file_url":"https://github.com/chancharikmitra/ccot/blob/HEAD/InstructBLIP-13b/InstructBLIP_MMBench.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bae18947b56f2be1"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"chancharikmitra/ccot","repo_kind":"official","path":"GPT-4V/Sphinx_bench.py","file_url":"https://github.com/chancharikmitra/ccot/blob/HEAD/GPT-4V/Sphinx_bench.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"b2a7f7ca0f25da4c","entry":"get_ans","repo":"chancharikmitra/ccot","repo_kind":"official","path":"GPT-4V/GPT4V_Whoops.py","file_url":"https://github.com/chancharikmitra/ccot/blob/HEAD/GPT-4V/GPT4V_Whoops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2a7f7ca0f25da4c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}