{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unleashing-the-potential-of-the-diffusion","title":"Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation","arxiv_id":"2410.02369","date":"2024-10-03","proceeding":null,"authors":["Muzhi Zhu","Yang Liu","Zekai Luo","Chenchen Jing","Hao Chen","Guangkai Xu","Xinlong Wang","Chunhua Shen"],"abstract":"The Diffusion Model has not only garnered noteworthy achievements in the realm of image generation but has also demonstrated its potential as an effective pretraining method utilizing unlabeled data. Drawing from the extensive potential unveiled by the Diffusion Model in both semantic correspondence and open vocabulary segmentation, our work initiates an investigation into employing the Latent Diffusion Model for Few-shot Semantic Segmentation. Recently, inspired by the in-context learning ability of large language models, Few-shot Semantic Segmentation has evolved into In-context Segmentation tasks, morphing into a crucial element in assessing generalist segmentation models. In this context, we concentrate on Few-shot Semantic Segmentation, establishing a solid foundation for the future development of a Diffusion-based generalist model for segmentation. Our initial focus lies in understanding how to facilitate interaction between the query image and the support image, resulting in the proposal of a KV fusion method within the self-attention framework. Subsequently, we delve deeper into optimizing the infusion of information from the support mask and simultaneously re-evaluating how to provide reasonable supervision from the query mask. Based on our analysis, we establish a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework and effectively utilizing the pre-training prior. Experimental results demonstrate that our method significantly outperforms the previous SOTA models in multiple settings.","url_abs":"https://arxiv.org/abs/2410.02369v3","url_pdf":"https://arxiv.org/pdf/2410.02369v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unleashing-the-potential-of-the-diffusion","repo_url":"https://github.com/aim-uofa/diffews","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"few-shot-image-segmentation","task_name":"Few-Shot Semantic Segmentation"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"semantic-correspondence","task_name":"Semantic correspondence"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"latent-diffusion-model","method_name":"Latent Diffusion Model"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/few-shot-semantic-segmentation-on-coco-20i-1","task":"Few-Shot Semantic Segmentation","dataset":"COCO-20i (1-shot)","model":"DiffewS (SD2.1)","rank_in_archive_order":6,"of":85,"metrics":{"Mean IoU":"52.2"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-semantic-segmentation-on-coco-20i-5","task":"Few-Shot Semantic Segmentation","dataset":"COCO-20i (5-shot)","model":"DiffewS(SD2.1)","rank_in_archive_order":5,"of":81,"metrics":{"Mean IoU":"60.7"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.02369","atlas_url":"https://app.syntology.ai/?focus=2410.02369","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.02369"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/aim-uofa/DiffewS","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aim-uofa/diffews","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":1,"ran_violates":2,"ran_fixture":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"28409ff8b4768607","entry":"get_test_dataloader","repo":"aim-uofa/DiffewS","repo_kind":"official","path":"train_tools/train_icl_multitask_nocrop_nearest_nshot_v3.py","file_url":"https://github.com/aim-uofa/DiffewS/blob/HEAD/train_tools/train_icl_multitask_nocrop_nearest_nshot_v3.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"28409ff8b4768607"}},{"code_sha256_prefix":"cd5d5067de871c97","entry":"isinstance_str","repo":"aim-uofa/diffews","repo_kind":"official","path":"diffews/models/unet_2d_condition.py","file_url":"https://github.com/aim-uofa/diffews/blob/HEAD/diffews/models/unet_2d_condition.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd5d5067de871c97"}},{"code_sha256_prefix":"4d43c39d84b84472","entry":"isinstance_str","repo":"aim-uofa/diffews","repo_kind":"official","path":"diffews/models/unet_2d_condition_attn.py","file_url":"https://github.com/aim-uofa/diffews/blob/HEAD/diffews/models/unet_2d_condition_attn.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d43c39d84b84472"}},{"code_sha256_prefix":"64d63260ebc6e238","entry":"pyramid_noise_like","repo":"aim-uofa/DiffewS","repo_kind":"official","path":"train_tools/train_icl_multitask_nocrop_nearest_nshot_v3.py","file_url":"https://github.com/aim-uofa/DiffewS/blob/HEAD/train_tools/train_icl_multitask_nocrop_nearest_nshot_v3.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"64d63260ebc6e238"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}