{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffusion-models-trained-with-large-data-are","title":"What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?","arxiv_id":"2403.06090","date":"2024-03-10","proceeding":null,"authors":["Guangkai Xu","Yongtao Ge","MingYu Liu","Chengxiang Fan","Kangyang Xie","Zhiyue Zhao","Hao Chen","Chunhua Shen"],"abstract":"Extensive pre-training with large data is indispensable for downstream geometry and semantic visual perception tasks. Thanks to large-scale text-to-image (T2I) pretraining, recent works show promising results by simply fine-tuning T2I diffusion models for dense perception tasks. However, several crucial design decisions in this process still lack comprehensive justification, encompassing the necessity of the multi-step stochastic diffusion mechanism, training strategy, inference ensemble strategy, and fine-tuning data quality. In this work, we conduct a thorough investigation into critical factors that affect transfer efficiency and performance when using diffusion priors. Our key findings are: 1) High-quality fine-tuning data is paramount for both semantic and geometry perception tasks. 2) The stochastic nature of diffusion models has a slightly negative impact on deterministic visual perception tasks. 3) Apart from fine-tuning the diffusion model with only latent space supervision, task-specific image-level supervision is beneficial to enhance fine-grained details. These observations culminate in the development of GenPercept, an effective deterministic one-step fine-tuning paradigm tailed for dense visual perception tasks. Different from the previous multi-step methods, our paradigm has a much faster inference speed, and can be seamlessly integrated with customized perception decoders and loss functions for image-level supervision, which is critical to improving the fine-grained details of predictions. Comprehensive experiments on diverse dense visual perceptual tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting, are performed to demonstrate the remarkable adaptability and effectiveness of our proposed method.","url_abs":"https://arxiv.org/abs/2403.06090v4","url_pdf":"https://arxiv.org/pdf/2403.06090v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffusion-models-trained-with-large-data-are","repo_url":"https://github.com/aim-uofa/genpercept","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"tasks":[{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"image-matting","task_name":"Image Matting"},{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"monocular-depth-estimation","task_name":"Monocular Depth Estimation"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"surface-normal-estimation","task_name":"Surface Normal Estimation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.06090","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.06090"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aim-uofa/genpercept","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"summary":{"ran_fixture":1,"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f9ec7d70add02b6f","entry":"HWC3","repo":"aim-uofa/genpercept","repo_kind":"official","path":"GenPercept_v1/hubconf.py","file_url":"https://github.com/aim-uofa/genpercept/blob/HEAD/GenPercept_v1/hubconf.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f9ec7d70add02b6f"}},{"code_sha256_prefix":"4c9c43aab3fc0cd6","entry":"align_scale_median_torch_batch","repo":"aim-uofa/genpercept","repo_kind":"official","path":"genpercept/losses/geometry_losses.py","file_url":"https://github.com/aim-uofa/genpercept/blob/HEAD/genpercept/losses/geometry_losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4c9c43aab3fc0cd6"}},{"code_sha256_prefix":"42fe451082384fe7","entry":"compute_scale_and_shift","repo":"aim-uofa/genpercept","repo_kind":"official","path":"genpercept/losses/geometry_losses.py","file_url":"https://github.com/aim-uofa/genpercept/blob/HEAD/genpercept/losses/geometry_losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"42fe451082384fe7"}},{"code_sha256_prefix":"64a28c95ec2c4293","entry":"gradient_log_loss","repo":"aim-uofa/genpercept","repo_kind":"official","path":"genpercept/losses/gradient_loss.py","file_url":"https://github.com/aim-uofa/genpercept/blob/HEAD/genpercept/losses/gradient_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"64a28c95ec2c4293"}},{"code_sha256_prefix":"fe38b7f1ddcaf1f2","entry":"gradient_loss","repo":"aim-uofa/genpercept","repo_kind":"official","path":"genpercept/losses/gradient_loss.py","file_url":"https://github.com/aim-uofa/genpercept/blob/HEAD/genpercept/losses/gradient_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"fe38b7f1ddcaf1f2"}},{"code_sha256_prefix":"2be26841f6fe76f2","entry":"match_scale_and_shift","repo":"aim-uofa/genpercept","repo_kind":"official","path":"genpercept/losses/geometry_losses.py","file_url":"https://github.com/aim-uofa/genpercept/blob/HEAD/genpercept/losses/geometry_losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2be26841f6fe76f2"}},{"code_sha256_prefix":"e8996630fe3501e1","entry":"reduction_batch_based","repo":"aim-uofa/genpercept","repo_kind":"official","path":"genpercept/losses/gradient_loss.py","file_url":"https://github.com/aim-uofa/genpercept/blob/HEAD/genpercept/losses/gradient_loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e8996630fe3501e1"}},{"code_sha256_prefix":"50fd68f6989503c6","entry":"resize_image","repo":"aim-uofa/genpercept","repo_kind":"official","path":"GenPercept_v1/hubconf.py","file_url":"https://github.com/aim-uofa/genpercept/blob/HEAD/GenPercept_v1/hubconf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"50fd68f6989503c6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}