{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffam-diffusion-based-adversarial-makeup","title":"DiffAM: Diffusion-based Adversarial Makeup Transfer for Facial Privacy Protection","arxiv_id":"2405.09882","date":"2024-05-16","proceeding":"CVPR 2024 1","authors":["Yuhao Sun","Lingyun Yu","Hongtao Xie","Jiaming Li","Yongdong Zhang"],"abstract":"With the rapid development of face recognition (FR) systems, the privacy of face images on social media is facing severe challenges due to the abuse of unauthorized FR systems. Some studies utilize adversarial attack techniques to defend against malicious FR systems by generating adversarial examples. However, the generated adversarial examples, i.e., the protected face images, tend to suffer from subpar visual quality and low transferability. In this paper, we propose a novel face protection approach, dubbed DiffAM, which leverages the powerful generative ability of diffusion models to generate high-quality protected face images with adversarial makeup transferred from reference images. To be specific, we first introduce a makeup removal module to generate non-makeup images utilizing a fine-tuned diffusion model with guidance of textual prompts in CLIP space. As the inverse process of makeup transfer, makeup removal can make it easier to establish the deterministic relationship between makeup domain and non-makeup domain regardless of elaborate text prompts. Then, with this relationship, a CLIP-based makeup loss along with an ensemble attack strategy is introduced to jointly guide the direction of adversarial makeup domain, achieving the generation of protected face images with natural-looking makeup and high black-box transferability. Extensive experiments demonstrate that DiffAM achieves higher visual quality and attack success rates with a gain of 12.98% under black-box setting compared with the state of the arts. The code will be available at https://github.com/HansSunY/DiffAM.","url_abs":"https://arxiv.org/abs/2405.09882v1","url_pdf":"https://arxiv.org/pdf/2405.09882v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffam-diffusion-based-adversarial-makeup","repo_url":"https://github.com/hanssuny/diffam","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"diffam-diffusion-based-adversarial-makeup","repo_url":"https://github.com/Recognito-Vision/Face-SDK-Linux-Demos","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"adversarial-attack","task_name":"Adversarial Attack"},{"task_slug":"face-recognition","task_name":"Face Recognition"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.09882","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.09882"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/HansSunY/DiffAM","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran":8,"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"c3a6b977022957cb","entry":"Normalize","repo":"HansSunY/DiffAM","repo_kind":"official","path":"models/ddpm/diffusion.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/models/ddpm/diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3a6b977022957cb"}},{"code_sha256_prefix":"501cb621a55f9607","entry":"cal_adv_loss","repo":"HansSunY/DiffAM","repo_kind":"official","path":"losses/id_loss.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/losses/id_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"501cb621a55f9607"}},{"code_sha256_prefix":"efae82b620750f99","entry":"cos_simi","repo":"HansSunY/DiffAM","repo_kind":"official","path":"losses/id_loss.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/losses/id_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efae82b620750f99"}},{"code_sha256_prefix":"1fc7a7721b881e1a","entry":"get_celeba_dataset","repo":"HansSunY/DiffAM","repo_kind":"official","path":"datasets/celeba_dataset.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/datasets/celeba_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1fc7a7721b881e1a"}},{"code_sha256_prefix":"360b55d77b3607a1","entry":"get_dataloader","repo":"HansSunY/DiffAM","repo_kind":"official","path":"datasets/data_utils.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/datasets/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"360b55d77b3607a1"}},{"code_sha256_prefix":"e41367ad14ff58fd","entry":"get_param_groups_and_shapes","repo":"HansSunY/DiffAM","repo_kind":"official","path":"models/improved_ddpm/fp16_util.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/models/improved_ddpm/fp16_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e41367ad14ff58fd"}},{"code_sha256_prefix":"cb49209c125de1b4","entry":"get_timestep_embedding","repo":"HansSunY/DiffAM","repo_kind":"official","path":"models/ddpm/diffusion.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/models/ddpm/diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb49209c125de1b4"}},{"code_sha256_prefix":"e20dd5102da3b050","entry":"make_master_params","repo":"HansSunY/DiffAM","repo_kind":"official","path":"models/improved_ddpm/fp16_util.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/models/improved_ddpm/fp16_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e20dd5102da3b050"}},{"code_sha256_prefix":"af251dae21d711a5","entry":"make_output_format","repo":"HansSunY/DiffAM","repo_kind":"official","path":"models/improved_ddpm/logger.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/models/improved_ddpm/logger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af251dae21d711a5"}},{"code_sha256_prefix":"3137073275f8c21a","entry":"nonlinearity","repo":"HansSunY/DiffAM","repo_kind":"official","path":"models/ddpm/diffusion.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/models/ddpm/diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3137073275f8c21a"}},{"code_sha256_prefix":"64fff1e30802b815","entry":"unflatten_master_params","repo":"HansSunY/DiffAM","repo_kind":"official","path":"models/improved_ddpm/fp16_util.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/models/improved_ddpm/fp16_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64fff1e30802b815"}},{"code_sha256_prefix":"414cb905a4aa4afe","entry":"get_dataset","repo":"HansSunY/DiffAM","repo_kind":"official","path":"datasets/data_utils.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/datasets/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"414cb905a4aa4afe"}},{"code_sha256_prefix":"e515a67f7f32e76d","entry":"mpi_weighted_mean","repo":"HansSunY/DiffAM","repo_kind":"official","path":"models/improved_ddpm/logger.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/models/improved_ddpm/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e515a67f7f32e76d"}},{"code_sha256_prefix":"956759c272bea487","entry":"profile","repo":"HansSunY/DiffAM","repo_kind":"official","path":"models/improved_ddpm/logger.py","file_url":"https://github.com/HansSunY/DiffAM/blob/HEAD/models/improved_ddpm/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"956759c272bea487"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}