{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/harnessing-diffusion-models-for-visual","title":"Harnessing Diffusion Models for Visual Perception with Meta Prompts","arxiv_id":"2312.14733","date":"2023-12-22","proceeding":null,"authors":["Qiang Wan","Zilong Huang","Bingyi Kang","Jiashi Feng","Li Zhang"],"abstract":"The issue of generative pretraining for vision models has persisted as a long-standing conundrum. At present, the text-to-image (T2I) diffusion model demonstrates remarkable proficiency in generating high-definition images matching textual inputs, a feat made possible through its pre-training on large-scale image-text pairs. This leads to a natural inquiry: can diffusion models be utilized to tackle visual perception tasks? In this paper, we propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. Our key insight is to introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. The effect of meta prompts are two-fold. First, as a direct replacement of the text embeddings in the T2I models, it can activate task-relevant features during feature extraction. Second, it will be used to re-arrange the extracted features to ensures that the model focuses on the most pertinent features for the task on hand. Additionally, we design a recurrent refinement training strategy that fully leverages the property of diffusion models, thereby yielding stronger visual features. Extensive experiments across various benchmarks validate the effectiveness of our approach. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes. Concurrently, the proposed method attains results comparable to the current state-of-the-art in semantic segmentation on ADE20K and pose estimation on COCO datasets, further exemplifying its robustness and versatility.","url_abs":"https://arxiv.org/abs/2312.14733v1","url_pdf":"https://arxiv.org/pdf/2312.14733v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"harnessing-diffusion-models-for-visual","repo_url":"https://github.com/fudan-zvg/meta-prompts","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"monocular-depth-estimation","task_name":"Monocular Depth Estimation"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/monocular-depth-estimation-on-kitti-eigen","task":"Monocular Depth Estimation","dataset":"KITTI Eigen split","model":"MetaPrompt-SD","rank_in_archive_order":11,"of":79,"metrics":{"Delta < 1.25":"0.981","Delta < 1.25^2":"0.998","Delta < 1.25^3":"1.000","RMSE":"1.928","RMSE log":"0.071","Sq Rel":"0.125","absolute relative error":"0.047"},"uses_additional_data":true},{"leaderboard":"/sota/monocular-depth-estimation-on-nyu-depth-v2","task":"Monocular Depth Estimation","dataset":"NYU-Depth V2","model":"MetaPrompt-SD","rank_in_archive_order":15,"of":85,"metrics":{"Delta < 1.25":"0.976","Delta < 1.25^2":"0.997","Delta < 1.25^3":"0.999","RMSE":"0.223","absolute relative error":"0.061","log 10":"0.027"},"uses_additional_data":true},{"leaderboard":"/sota/pose-estimation-on-coco","task":"Pose Estimation","dataset":"COCO (Common Objects in Context)","model":"MetaPrompt-SD","rank_in_archive_order":2,"of":10,"metrics":{"AP":"79.0"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"MetaPrompt-SD","rank_in_archive_order":37,"of":235,"metrics":{"Validation mIoU":"56.8"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-cityscapes","task":"Semantic Segmentation","dataset":"Cityscapes test","model":"MetaPrompt-SD","rank_in_archive_order":2,"of":105,"metrics":{"Mean IoU (class)":"86.2"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-cityscapes-val","task":"Semantic Segmentation","dataset":"Cityscapes val","model":"MetaPrompt-SD","rank_in_archive_order":3,"of":99,"metrics":{"mIoU":"87.1"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.14733","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.14733"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fudan-zvg/meta-prompts","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":2,"ran":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"424012cb37b31172","entry":"default","repo":"fudan-zvg/meta-prompts","repo_kind":"official","path":"meta_prompts/models.py","file_url":"https://github.com/fudan-zvg/meta-prompts/blob/HEAD/meta_prompts/models.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"424012cb37b31172"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"fudan-zvg/meta-prompts","repo_kind":"official","path":"meta_prompts/models.py","file_url":"https://github.com/fudan-zvg/meta-prompts/blob/HEAD/meta_prompts/models.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"6907f72868217c02","entry":"get_num_layer_for_swin","repo":"fudan-zvg/meta-prompts","repo_kind":"official","path":"depth/models_depth/optimizer.py","file_url":"https://github.com/fudan-zvg/meta-prompts/blob/HEAD/depth/models_depth/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6907f72868217c02"}},{"code_sha256_prefix":"9a299fe5ae09e407","entry":"uniq","repo":"fudan-zvg/meta-prompts","repo_kind":"official","path":"meta_prompts/models.py","file_url":"https://github.com/fudan-zvg/meta-prompts/blob/HEAD/meta_prompts/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a299fe5ae09e407"}},{"code_sha256_prefix":"b700fb0a3496b255","entry":"cosine_scheduler","repo":"fudan-zvg/meta-prompts","repo_kind":"official","path":"depth/utils.py","file_url":"https://github.com/fudan-zvg/meta-prompts/blob/HEAD/depth/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b700fb0a3496b255"}},{"code_sha256_prefix":"ba1356e8ceb654d2","entry":"get_grad_norm_","repo":"fudan-zvg/meta-prompts","repo_kind":"official","path":"depth/utils.py","file_url":"https://github.com/fudan-zvg/meta-prompts/blob/HEAD/depth/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba1356e8ceb654d2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}