{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ip-adapter-text-compatible-image-prompt","title":"IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models","arxiv_id":"2308.06721","date":"2023-08-13","proceeding":null,"authors":["Hu Ye","Jun Zhang","Sibo Liu","Xiao Han","Wei Yang"],"abstract":"Recent years have witnessed the strong power of large text-to-image diffusion models for the impressive generative capability to create high-fidelity images. However, it is very tricky to generate desired images using only text prompt as it often involves complex prompt engineering. An alternative to text prompt is image prompt, as the saying goes: \"an image is worth a thousand words\". Although existing methods of direct fine-tuning from pretrained models are effective, they require large computing resources and are not compatible with other base models, text prompt, and structural controls. In this paper, we present IP-Adapter, an effective and lightweight adapter to achieve image prompt capability for the pretrained text-to-image diffusion models. The key design of our IP-Adapter is decoupled cross-attention mechanism that separates cross-attention layers for text features and image features. Despite the simplicity of our method, an IP-Adapter with only 22M parameters can achieve comparable or even better performance to a fully fine-tuned image prompt model. As we freeze the pretrained diffusion model, the proposed IP-Adapter can be generalized not only to other custom models fine-tuned from the same base model, but also to controllable generation using existing controllable tools. With the benefit of the decoupled cross-attention strategy, the image prompt can also work well with the text prompt to achieve multimodal image generation. The project page is available at \\url{https://ip-adapter.github.io}.","url_abs":"https://arxiv.org/abs/2308.06721v1","url_pdf":"https://arxiv.org/pdf/2308.06721v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ip-adapter-text-compatible-image-prompt","repo_url":"https://github.com/fhshen2022/prunerepaint","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"ip-adapter-text-compatible-image-prompt","repo_url":"https://github.com/liuxiaoyu1104/smartcontrol","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"ip-adapter-text-compatible-image-prompt","repo_url":"https://github.com/tencent-ailab/ip-adapter","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"ip-adapter-text-compatible-image-prompt","repo_url":"https://github.com/mindspore-lab/mindone","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"tasks":[{"task_slug":"diffusion-personalization-tuning-free","task_name":"Diffusion Personalization Tuning Free"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"personalized-image-generation","task_name":"Personalized Image Generation"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/diffusion-personalization-tuning-free-on","task":"Diffusion Personalization Tuning Free","dataset":"AgeDB","model":"IP-Adapter-FaceID-Plus","rank_in_archive_order":3,"of":7,"metrics":{"Cosine Similarity":"0.600","FID":"11.817","LPIPS":"0.384"},"uses_additional_data":false},{"leaderboard":"/sota/diffusion-personalization-tuning-free-on","task":"Diffusion Personalization Tuning Free","dataset":"AgeDB","model":"IP-Adapter-FaceID-PlusV2","rank_in_archive_order":4,"of":7,"metrics":{"Cosine Similarity":"0.572","FID":"10.798","LPIPS":"0.429"},"uses_additional_data":false},{"leaderboard":"/sota/diffusion-personalization-tuning-free-on","task":"Diffusion Personalization Tuning Free","dataset":"AgeDB","model":"IP-Adapter-FaceID (SDXL)","rank_in_archive_order":5,"of":7,"metrics":{"Cosine Similarity":"0.535","FID":"24.105","LPIPS":"0.462"},"uses_additional_data":false},{"leaderboard":"/sota/personalized-image-generation-on-dreambench","task":"Personalized Image Generation","dataset":"DreamBooth","model":"IP-Adapter ViT-G SDXL v1.0","rank_in_archive_order":2,"of":7,"metrics":{"Concept Preservation (CP)":"0.593","Overall (CP * PF)":"0.380","Prompt Following (PF)":"0.640"},"uses_additional_data":false},{"leaderboard":"/sota/personalized-image-generation-on-dreambench","task":"Personalized Image Generation","dataset":"DreamBooth","model":"IP-Adapter-Plus ViT-H SDXL v1.0","rank_in_archive_order":5,"of":7,"metrics":{"Concept Preservation (CP)":"0.833","Overall (CP * PF)":"0.344","Prompt Following (PF)":"0.413"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.06721","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.06721"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liuxiaoyu1104/smartcontrol","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fhshen2022/prunerepaint","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-lab/mindone","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tencent-ailab/ip-adapter","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":2,"unverified":5},"by_repo_kind":{"listed":{"samples":8,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5105747c2711b1cb","entry":"FeedForward","repo":"tencent-ailab/ip-adapter","repo_kind":"listed","path":"ip_adapter/resampler.py","file_url":"https://github.com/tencent-ailab/ip-adapter/blob/HEAD/ip_adapter/resampler.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5105747c2711b1cb"}},{"code_sha256_prefix":"01c0e92550c3a1ca","entry":"collate_fn","repo":"tencent-ailab/ip-adapter","repo_kind":"listed","path":"tutorial_train.py","file_url":"https://github.com/tencent-ailab/ip-adapter/blob/HEAD/tutorial_train.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01c0e92550c3a1ca"}},{"code_sha256_prefix":"1e0487828d53d63e","entry":"collate_fn","repo":"tencent-ailab/ip-adapter","repo_kind":"listed","path":"tutorial_train_faceid.py","file_url":"https://github.com/tencent-ailab/ip-adapter/blob/HEAD/tutorial_train_faceid.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e0487828d53d63e"}},{"code_sha256_prefix":"06d3e795b3014ef1","entry":"hook_fn","repo":"tencent-ailab/ip-adapter","repo_kind":"listed","path":"ip_adapter/utils.py","file_url":"https://github.com/tencent-ailab/ip-adapter/blob/HEAD/ip_adapter/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06d3e795b3014ef1"}},{"code_sha256_prefix":"230588e8f1237737","entry":"masked_mean","repo":"tencent-ailab/ip-adapter","repo_kind":"listed","path":"ip_adapter/resampler.py","file_url":"https://github.com/tencent-ailab/ip-adapter/blob/HEAD/ip_adapter/resampler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"230588e8f1237737"}},{"code_sha256_prefix":"cf95fb65ebfe64b4","entry":"register_cross_attention_hook","repo":"tencent-ailab/ip-adapter","repo_kind":"listed","path":"ip_adapter/utils.py","file_url":"https://github.com/tencent-ailab/ip-adapter/blob/HEAD/ip_adapter/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf95fb65ebfe64b4"}},{"code_sha256_prefix":"4cb2e2a2ca0bec9f","entry":"reshape_tensor","repo":"tencent-ailab/ip-adapter","repo_kind":"listed","path":"ip_adapter/resampler.py","file_url":"https://github.com/tencent-ailab/ip-adapter/blob/HEAD/ip_adapter/resampler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb2e2a2ca0bec9f"}},{"code_sha256_prefix":"25837bff8f005465","entry":"upscale","repo":"tencent-ailab/ip-adapter","repo_kind":"listed","path":"ip_adapter/utils.py","file_url":"https://github.com/tencent-ailab/ip-adapter/blob/HEAD/ip_adapter/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25837bff8f005465"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}