{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/got-unleashing-reasoning-capability-of","title":"GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editing","arxiv_id":"2503.10639","date":"2025-03-13","proceeding":null,"authors":["Rongyao Fang","Chengqi Duan","Kun Wang","Linjiang Huang","Hao Li","Shilin Yan","Hao Tian","Xingyu Zeng","Rui Zhao","Jifeng Dai","Xihui Liu","Hongsheng Li"],"abstract":"Current image generation and editing methods primarily process textual prompts as direct inputs without reasoning about visual composition and explicit operations. We present Generation Chain-of-Thought (GoT), a novel paradigm that enables generation and editing through an explicit language reasoning process before outputting images. This approach transforms conventional text-to-image generation and editing into a reasoning-guided framework that analyzes semantic relationships and spatial arrangements. We define the formulation of GoT and construct large-scale GoT datasets containing over 9M samples with detailed reasoning chains capturing semantic-spatial relationships. To leverage the advantages of GoT, we implement a unified framework that integrates Qwen2.5-VL for reasoning chain generation with an end-to-end diffusion model enhanced by our novel Semantic-Spatial Guidance Module. Experiments show our GoT framework achieves excellent performance on both generation and editing tasks, with significant improvements over baselines. Additionally, our approach enables interactive visual generation, allowing users to explicitly modify reasoning steps for precise image adjustments. GoT pioneers a new direction for reasoning-driven visual generation and editing, producing images that better align with human intent. To facilitate future research, we make our datasets, code, and pretrained models publicly available at https://github.com/rongyaofang/GoT.","url_abs":"https://arxiv.org/abs/2503.10639v1","url_pdf":"https://arxiv.org/pdf/2503.10639v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"got-unleashing-reasoning-capability-of","repo_url":"https://github.com/rongyaofang/got","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"multimodal-large-language-model","task_name":"Multimodal Large Language Model"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.10639","atlas_url":"https://app.syntology.ai/?focus=2503.10639","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.10639"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rongyaofang/got","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3a7ada9ba38fa217","entry":"get_processor","repo":"rongyaofang/got","repo_kind":"official","path":"got/processer/qwen25_vl_processor.py","file_url":"https://github.com/rongyaofang/got/blob/HEAD/got/processer/qwen25_vl_processor.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a7ada9ba38fa217"}},{"code_sha256_prefix":"d744c9727511497a","entry":"parse_coordinates_colors","repo":"rongyaofang/got","repo_kind":"official","path":"got/models/utils.py","file_url":"https://github.com/rongyaofang/got/blob/HEAD/got/models/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d744c9727511497a"}},{"code_sha256_prefix":"8bbb8a75f0c9fe32","entry":"remove_mismatched_weights","repo":"rongyaofang/got","repo_kind":"official","path":"got/models/utils.py","file_url":"https://github.com/rongyaofang/got/blob/HEAD/got/models/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8bbb8a75f0c9fe32"}},{"code_sha256_prefix":"2a00c97debc15a3e","entry":"get_peft_model_without_resize_embedding","repo":"rongyaofang/got","repo_kind":"official","path":"got/models/peft_models.py","file_url":"https://github.com/rongyaofang/got/blob/HEAD/got/models/peft_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a00c97debc15a3e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}