{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/instruct-clip-improving-instruction-guided","title":"Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning","arxiv_id":"2503.18406","date":"2025-03-24","proceeding":"CVPR 2025 1","authors":["Sherry X. Chen","Misha Sra","Pradeep Sen"],"abstract":"Although natural language instructions offer an intuitive way to guide automated image editing, deep-learning models often struggle to achieve high-quality results, largely due to challenges in creating large, high-quality training datasets. Previous work has typically relied on text-toimage (T2I) generative models to produce pairs of original and edited images that simulate the input/output of an instruction-guided image-editing model. However, these image pairs often fail to align with the specified edit instructions due to the limitations of T2I models, which negatively impacts models trained on such datasets. To address this, we present Instruct-CLIP, a self-supervised method that learns the semantic changes between original and edited images to refine and better align the instructions in existing datasets. Furthermore, we adapt Instruct-CLIP to handle noisy latent images and diffusion timesteps so that it can be used to train latent diffusion models (LDMs) [19] and efficiently enforce alignment between the edit instruction and the image changes in latent space at any step of the diffusion pipeline. We use Instruct-CLIP to correct the InstructPix2Pix dataset and get over 120K refined samples we then use to fine-tune their model, guided by our novel Instruct-CLIP-based loss function. The resulting model can produce edits that are more aligned with the given instructions. Our code and dataset are available at https://github.com/SherryXTChen/Instruct-CLIP.git.","url_abs":"https://arxiv.org/abs/2503.18406v1","url_pdf":"https://arxiv.org/pdf/2503.18406v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"instruct-clip-improving-instruction-guided","repo_url":"https://github.com/sherryxtchen/instruct-clip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.18406","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.18406"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/SherryXTChen/Instruct-CLIP","reach":null}],"summary":{"ran_draft_wrong":1,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"86961ea7f4dc10d3","entry":"get_features","repo":"SherryXTChen/Instruct-CLIP","repo_kind":"official","path":"model.py","file_url":"https://github.com/SherryXTChen/Instruct-CLIP/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"86961ea7f4dc10d3"}},{"code_sha256_prefix":"f838567d1f5d48f7","entry":"normalize","repo":"SherryXTChen/Instruct-CLIP","repo_kind":"official","path":"model.py","file_url":"https://github.com/SherryXTChen/Instruct-CLIP/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f838567d1f5d48f7"}},{"code_sha256_prefix":"5d3d363a4cf3a26d","entry":"InstructCLIP","repo":"SherryXTChen/Instruct-CLIP","repo_kind":"official","path":"model.py","file_url":"https://github.com/SherryXTChen/Instruct-CLIP/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d3d363a4cf3a26d"}},{"code_sha256_prefix":"e75437285f68a188","entry":"LDDinov2","repo":"SherryXTChen/Instruct-CLIP","repo_kind":"official","path":"model.py","file_url":"https://github.com/SherryXTChen/Instruct-CLIP/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e75437285f68a188"}},{"code_sha256_prefix":"8ddbaf4ca2471a91","entry":"Model","repo":"SherryXTChen/Instruct-CLIP","repo_kind":"official","path":"model.py","file_url":"https://github.com/SherryXTChen/Instruct-CLIP/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ddbaf4ca2471a91"}},{"code_sha256_prefix":"a3c6930dab1d7260","entry":"TextDecoder","repo":"SherryXTChen/Instruct-CLIP","repo_kind":"official","path":"model.py","file_url":"https://github.com/SherryXTChen/Instruct-CLIP/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a3c6930dab1d7260"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}