{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cogcom-train-large-vision-language-models","title":"CogCoM: Train Large Vision-Language Models Diving into Details through Chain of Manipulations","arxiv_id":"2402.04236","date":"2024-02-06","proceeding":null,"authors":["Ji Qi","Ming Ding","Weihan Wang","Yushi Bai","Qingsong Lv","Wenyi Hong","Bin Xu","Lei Hou","Juanzi Li","Yuxiao Dong","Jie Tang"],"abstract":"Vision-Language Models (VLMs) have demonstrated their broad effectiveness thanks to extensive training in aligning visual instructions to responses. However, such training of conclusive alignment leads models to ignore essential visual reasoning, further resulting in failures in meticulous visual problems and unfaithful responses. Drawing inspiration from human cognition in solving visual problems (e.g., marking, zoom in), this paper introduces Chain of Manipulations, a mechanism that enables VLMs to solve problems step-by-step with evidence. After training, models can solve various visual problems by eliciting intrinsic manipulations (e.g., grounding, zoom in) with results (e.g., boxes, image) actively without involving external tools, while also allowing users to trace error causes. We study the roadmap to implement this mechanism, including (1) a flexible design of manipulations upon extensive analysis, (2) an efficient automated data generation pipeline, (3) a compatible VLM architecture capable of multi-turn multi-image, and (4) a model training process for versatile capabilities. With the design, we also manually annotate 6K high-quality samples for the challenging graphical mathematical problems. Our trained model, \\textbf{CogCoM}, equipped with this mechanism with 17B parameters achieves state-of-the-art performance across 9 benchmarks from 4 categories, demonstrating the effectiveness while preserving the interpretability. Our code, model weights, and collected data are publicly available at https://github.com/THUDM/CogCoM.","url_abs":"https://arxiv.org/abs/2402.04236v2","url_pdf":"https://arxiv.org/pdf/2402.04236v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cogcom-train-large-vision-language-models","repo_url":"https://github.com/thudm/cogcom","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.04236","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.04236"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thudm/cogcom","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":11},"by_repo_kind":{"official":{"samples":11,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"408cf9b8b5aeb28a","entry":"build_tree","repo":"thudm/cogcom","repo_kind":"official","path":"cogcom/utils/com_dataset.py","file_url":"https://github.com/thudm/cogcom/blob/HEAD/cogcom/utils/com_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"408cf9b8b5aeb28a"}},{"code_sha256_prefix":"546aa05ddb9110de","entry":"build_tree","repo":"thudm/cogcom","repo_kind":"official","path":"cogcom/utils/com_utils.py","file_url":"https://github.com/thudm/cogcom/blob/HEAD/cogcom/utils/com_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"546aa05ddb9110de"}},{"code_sha256_prefix":"5b78db65a8720067","entry":"data_collator_com","repo":"thudm/cogcom","repo_kind":"official","path":"cogcom/finetune.py","file_url":"https://github.com/thudm/cogcom/blob/HEAD/cogcom/finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5b78db65a8720067"}},{"code_sha256_prefix":"390b49491af0d867","entry":"find_paths","repo":"thudm/cogcom","repo_kind":"official","path":"cogcom/utils/com_dataset.py","file_url":"https://github.com/thudm/cogcom/blob/HEAD/cogcom/utils/com_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"390b49491af0d867"}},{"code_sha256_prefix":"4f51dc745dbe5e37","entry":"get_text_by_box","repo":"thudm/cogcom","repo_kind":"official","path":"cogcom/utils/grounding_parser.py","file_url":"https://github.com/thudm/cogcom/blob/HEAD/cogcom/utils/grounding_parser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4f51dc745dbe5e37"}},{"code_sha256_prefix":"9dbd5a41263c5a75","entry":"override_dist_dtype_device_args","repo":"thudm/cogcom","repo_kind":"official","path":"cogcom/models/cogcom_model.py","file_url":"https://github.com/thudm/cogcom/blob/HEAD/cogcom/models/cogcom_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9dbd5a41263c5a75"}},{"code_sha256_prefix":"7e55993d5a3ea8b2","entry":"process_image","repo":"thudm/cogcom","repo_kind":"official","path":"cogcom/utils/chat.py","file_url":"https://github.com/thudm/cogcom/blob/HEAD/cogcom/utils/chat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7e55993d5a3ea8b2"}},{"code_sha256_prefix":"dc039cc4ef751f3a","entry":"process_image_without_resize","repo":"thudm/cogcom","repo_kind":"official","path":"cogcom/demo/web_demo.py","file_url":"https://github.com/thudm/cogcom/blob/HEAD/cogcom/demo/web_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dc039cc4ef751f3a"}},{"code_sha256_prefix":"a001d45341e98d9d","entry":"refine_box","repo":"thudm/cogcom","repo_kind":"official","path":"cogcom/utils/grounding_parser.py","file_url":"https://github.com/thudm/cogcom/blob/HEAD/cogcom/utils/grounding_parser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a001d45341e98d9d"}},{"code_sha256_prefix":"dd78601e3d5fa6b1","entry":"unrefine_box","repo":"thudm/cogcom","repo_kind":"official","path":"cogcom/utils/grounding_parser.py","file_url":"https://github.com/thudm/cogcom/blob/HEAD/cogcom/utils/grounding_parser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dd78601e3d5fa6b1"}},{"code_sha256_prefix":"10c628be05905d42","entry":"update_mems","repo":"thudm/cogcom","repo_kind":"official","path":"cogcom/models/com_memory.py","file_url":"https://github.com/thudm/cogcom/blob/HEAD/cogcom/models/com_memory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"10c628be05905d42"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}