{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/attention-calibration-for-disentangled-text","title":"Attention Calibration for Disentangled Text-to-Image Personalization","arxiv_id":"2403.18551","date":"2024-03-27","proceeding":"CVPR 2024 1","authors":["Yanbing Zhang","Mengping Yang","Qin Zhou","Zhe Wang"],"abstract":"Recent thrilling progress in large-scale text-to-image (T2I) models has unlocked unprecedented synthesis quality of AI-generated content (AIGC) including image generation, 3D and video composition. Further, personalized techniques enable appealing customized production of a novel concept given only several images as reference. However, an intriguing problem persists: Is it possible to capture multiple, novel concepts from one single reference image? In this paper, we identify that existing approaches fail to preserve visual consistency with the reference image and eliminate cross-influence from concepts. To alleviate this, we propose an attention calibration mechanism to improve the concept-level understanding of the T2I model. Specifically, we first introduce new learnable modifiers bound with classes to capture attributes of multiple concepts. Then, the classes are separated and strengthened following the activation of the cross-attention operation, ensuring comprehensive and self-contained concepts. Additionally, we suppress the attention activation of different classes to mitigate mutual influence among concepts. Together, our proposed method, dubbed DisenDiff, can learn disentangled multiple concepts from one single image and produce novel customized images with learned concepts. We demonstrate that our method outperforms the current state of the art in both qualitative and quantitative evaluations. More importantly, our proposed techniques are compatible with LoRA and inpainting pipelines, enabling more interactive experiences.","url_abs":"https://arxiv.org/abs/2403.18551v2","url_pdf":"https://arxiv.org/pdf/2403.18551v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"attention-calibration-for-disentangled-text","repo_url":"https://github.com/monalissaa/disendiff","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"novel-concepts","task_name":"Novel Concepts"}],"methods":[{"method_slug":"pixel-prediction","method_name":"Inpainting"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.18551","atlas_url":"https://app.syntology.ai/?focus=2403.18551","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.18551"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/monalissaa/disendiff","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Monalissaa/DisenDiff","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2ba0d66eaa1687b5","entry":"CDAttentionStore","repo":"Monalissaa/DisenDiff","repo_kind":"official","path":"src/attention_controllers.py","file_url":"https://github.com/Monalissaa/DisenDiff/blob/HEAD/src/attention_controllers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ba0d66eaa1687b5"}},{"code_sha256_prefix":"31b8b7196c34d874","entry":"collate_fn","repo":"monalissaa/disendiff","repo_kind":"official","path":"src/diffusers_data_pipeline.py","file_url":"https://github.com/monalissaa/disendiff/blob/HEAD/src/diffusers_data_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31b8b7196c34d874"}},{"code_sha256_prefix":"9a06e6a953d84b4d","entry":"preprocess","repo":"monalissaa/disendiff","repo_kind":"official","path":"src/diffusers_data_pipeline.py","file_url":"https://github.com/monalissaa/disendiff/blob/HEAD/src/diffusers_data_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a06e6a953d84b4d"}},{"code_sha256_prefix":"3ac63e51754d948b","entry":"text_under_image","repo":"monalissaa/disendiff","repo_kind":"official","path":"sample.py","file_url":"https://github.com/monalissaa/disendiff/blob/HEAD/sample.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ac63e51754d948b"}},{"code_sha256_prefix":"7ecaa1ab867d461d","entry":"build_model","repo":"monalissaa/disendiff","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/monalissaa/disendiff/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ecaa1ab867d461d"}},{"code_sha256_prefix":"844fb2cc091364b4","entry":"gdupdateWexact","repo":"monalissaa/disendiff","repo_kind":"official","path":"src/composenW.py","file_url":"https://github.com/monalissaa/disendiff/blob/HEAD/src/composenW.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"844fb2cc091364b4"}},{"code_sha256_prefix":"2f00b39db5a81466","entry":"import_model_class_from_model_name_or_path","repo":"monalissaa/disendiff","repo_kind":"official","path":"src/diffusers_training.py","file_url":"https://github.com/monalissaa/disendiff/blob/HEAD/src/diffusers_training.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f00b39db5a81466"}},{"code_sha256_prefix":"3a8e786a49a73ff3","entry":"parse_args","repo":"monalissaa/disendiff","repo_kind":"official","path":"src/diffusers_training.py","file_url":"https://github.com/monalissaa/disendiff/blob/HEAD/src/diffusers_training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a8e786a49a73ff3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}