{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/high-fidelity-person-centric-subject-to-image","title":"High-fidelity Person-centric Subject-to-Image Synthesis","arxiv_id":"2311.10329","date":"2023-11-17","proceeding":"CVPR 2024 1","authors":["Yibin Wang","Weizhong Zhang","Jianwei Zheng","Cheng Jin"],"abstract":"Current subject-driven image generation methods encounter significant challenges in person-centric image generation. The reason is that they learn the semantic scene and person generation by fine-tuning a common pre-trained diffusion, which involves an irreconcilable training imbalance. Precisely, to generate realistic persons, they need to sufficiently tune the pre-trained model, which inevitably causes the model to forget the rich semantic scene prior and makes scene generation over-fit to the training data. Moreover, even with sufficient fine-tuning, these methods can still not generate high-fidelity persons since joint learning of the scene and person generation also lead to quality compromise. In this paper, we propose Face-diffuser, an effective collaborative generation pipeline to eliminate the above training imbalance and quality compromise. Specifically, we first develop two specialized pre-trained diffusion models, i.e., Text-driven Diffusion Model (TDM) and Subject-augmented Diffusion Model (SDM), for scene and person generation, respectively. The sampling process is divided into three sequential stages, i.e., semantic scene construction, subject-scene fusion, and subject enhancement. The first and last stages are performed by TDM and SDM respectively. The subject-scene fusion stage, that is the collaboration achieved through a novel and highly effective mechanism, Saliency-adaptive Noise Fusion (SNF). Specifically, it is based on our key observation that there exists a robust link between classifier-free guidance responses and the saliency of generated images. In each time step, SNF leverages the unique strengths of each model and allows for the spatial blending of predicted noises from both models automatically in a saliency-aware manner. Extensive experiments confirm the impressive effectiveness and robustness of the Face-diffuser.","url_abs":"https://arxiv.org/abs/2311.10329v5","url_pdf":"https://arxiv.org/pdf/2311.10329v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"high-fidelity-person-centric-subject-to-image","repo_url":"https://github.com/codegoat24/face-diffuser","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"scene-generation","task_name":"Scene Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.10329","atlas_url":"https://app.syntology.ai/?focus=2311.10329","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.10329"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/codegoat24/face-diffuser","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":5},"by_repo_kind":{"official":{"samples":11,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ba203f24d15e031a","entry":"get_data_loader","repo":"codegoat24/face-diffuser","repo_kind":"official","path":"facediffuser/data.py","file_url":"https://github.com/codegoat24/face-diffuser/blob/HEAD/facediffuser/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba203f24d15e031a"}},{"code_sha256_prefix":"addaaf66c3ab7352","entry":"get_test_transforms","repo":"codegoat24/face-diffuser","repo_kind":"official","path":"facediffuser/transforms.py","file_url":"https://github.com/codegoat24/face-diffuser/blob/HEAD/facediffuser/transforms.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"addaaf66c3ab7352"}},{"code_sha256_prefix":"c036796b62b5952d","entry":"get_train_transforms","repo":"codegoat24/face-diffuser","repo_kind":"official","path":"facediffuser/transforms.py","file_url":"https://github.com/codegoat24/face-diffuser/blob/HEAD/facediffuser/transforms.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c036796b62b5952d"}},{"code_sha256_prefix":"15ed5aec7f6bd270","entry":"get_train_transforms_with_segmap","repo":"codegoat24/face-diffuser","repo_kind":"official","path":"facediffuser/transforms.py","file_url":"https://github.com/codegoat24/face-diffuser/blob/HEAD/facediffuser/transforms.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15ed5aec7f6bd270"}},{"code_sha256_prefix":"817e9d69c2cd23d3","entry":"prepare_image_token_idx","repo":"codegoat24/face-diffuser","repo_kind":"official","path":"facediffuser/data.py","file_url":"https://github.com/codegoat24/face-diffuser/blob/HEAD/facediffuser/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"817e9d69c2cd23d3"}},{"code_sha256_prefix":"ff242689fdee9a2e","entry":"scatter_object_embeddings","repo":"codegoat24/face-diffuser","repo_kind":"official","path":"facediffuser/model.py","file_url":"https://github.com/codegoat24/face-diffuser/blob/HEAD/facediffuser/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff242689fdee9a2e"}},{"code_sha256_prefix":"33f63d47933a8a81","entry":"collate_fn","repo":"codegoat24/face-diffuser","repo_kind":"official","path":"facediffuser/data.py","file_url":"https://github.com/codegoat24/face-diffuser/blob/HEAD/facediffuser/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33f63d47933a8a81"}},{"code_sha256_prefix":"2fa8e59bc1db947b","entry":"fuse_object_embeddings","repo":"codegoat24/face-diffuser","repo_kind":"official","path":"facediffuser/model.py","file_url":"https://github.com/codegoat24/face-diffuser/blob/HEAD/facediffuser/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fa8e59bc1db947b"}},{"code_sha256_prefix":"b097d3c89f64264b","entry":"parse_args","repo":"codegoat24/face-diffuser","repo_kind":"official","path":"facediffuser/util.py","file_url":"https://github.com/codegoat24/face-diffuser/blob/HEAD/facediffuser/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b097d3c89f64264b"}},{"code_sha256_prefix":"b875306d3e07bc84","entry":"stable_diffusion_call_with_references_delayed_conditioning","repo":"codegoat24/face-diffuser","repo_kind":"official","path":"facediffuser/pipeline.py","file_url":"https://github.com/codegoat24/face-diffuser/blob/HEAD/facediffuser/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b875306d3e07bc84"}},{"code_sha256_prefix":"0a5a54048f832e12","entry":"unet_store_cross_attention_scores","repo":"codegoat24/face-diffuser","repo_kind":"official","path":"facediffuser/model.py","file_url":"https://github.com/codegoat24/face-diffuser/blob/HEAD/facediffuser/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a5a54048f832e12"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}