{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/guiding-instruction-based-image-editing-via","title":"Guiding Instruction-based Image Editing via Multimodal Large Language Models","arxiv_id":"2309.17102","date":"2023-09-29","proceeding":null,"authors":["Tsu-Jui Fu","Wenze Hu","Xianzhi Du","William Yang Wang","Yinfei Yang","Zhe Gan"],"abstract":"Instruction-based image editing improves the controllability and flexibility of image manipulation via natural commands without elaborate descriptions or regional masks. However, human instructions are sometimes too brief for current methods to capture and follow. Multimodal large language models (MLLMs) show promising capabilities in cross-modal understanding and visual-aware response generation via LMs. We investigate how MLLMs facilitate edit instructions and present MLLM-Guided Image Editing (MGIE). MGIE learns to derive expressive instructions and provides explicit guidance. The editing model jointly captures this visual imagination and performs manipulation through end-to-end training. We evaluate various aspects of Photoshop-style modification, global photo optimization, and local editing. Extensive experimental results demonstrate that expressive instructions are crucial to instruction-based image editing, and our MGIE can lead to a notable improvement in automatic metrics and human evaluation while maintaining competitive inference efficiency.","url_abs":"https://arxiv.org/abs/2309.17102v2","url_pdf":"https://arxiv.org/pdf/2309.17102v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"guiding-instruction-based-image-editing-via","repo_url":"https://github.com/apple/ml-mgie","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"guiding-instruction-based-image-editing-via","repo_url":"https://github.com/tsujuifu/pytorch_mgie","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-manipulation","task_name":"Image Manipulation"},{"task_slug":"response-generation","task_name":"Response Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.17102","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.17102"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/apple/ml-mgie","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tsujuifu/pytorch_mgie","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran_fixture":1,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"653e2b2a3dcc281d","entry":"b2f","repo":"apple/ml-mgie","repo_kind":"official","path":"mgie_train.py","file_url":"https://github.com/apple/ml-mgie/blob/HEAD/mgie_train.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"653e2b2a3dcc281d"}},{"code_sha256_prefix":"16c75ca40ba740d6","entry":"img2npy","repo":"apple/ml-mgie","repo_kind":"official","path":"mgie_train.py","file_url":"https://github.com/apple/ml-mgie/blob/HEAD/mgie_train.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"16c75ca40ba740d6"}},{"code_sha256_prefix":"7760f26e090b6580","entry":"resize","repo":"apple/ml-mgie","repo_kind":"official","path":"mgie_train.py","file_url":"https://github.com/apple/ml-mgie/blob/HEAD/mgie_train.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7760f26e090b6580"}},{"code_sha256_prefix":"5b0fc6b892b3d1e7","entry":"EditMapper","repo":"apple/ml-mgie","repo_kind":"official","path":"mgie_llava.py","file_url":"https://github.com/apple/ml-mgie/blob/HEAD/mgie_llava.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5b0fc6b892b3d1e7"}},{"code_sha256_prefix":"22e57ed82fd754ec","entry":"LlavaLlamaForCausalLM","repo":"apple/ml-mgie","repo_kind":"official","path":"mgie_llava.py","file_url":"https://github.com/apple/ml-mgie/blob/HEAD/mgie_llava.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"22e57ed82fd754ec"}},{"code_sha256_prefix":"1ccdac673c8f650c","entry":"LlavaLlamaModel","repo":"apple/ml-mgie","repo_kind":"official","path":"mgie_llava.py","file_url":"https://github.com/apple/ml-mgie/blob/HEAD/mgie_llava.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1ccdac673c8f650c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}