{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/datasetdm-synthesizing-data-with-perception-1","title":"DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models","arxiv_id":"2308.06160","date":"2023-08-11","proceeding":"NeurIPS 2023 11","authors":["Weijia Wu","Yuzhong Zhao","Hao Chen","YuChao Gu","Rui Zhao","Yefei He","Hong Zhou","Mike Zheng Shou","Chunhua Shen"],"abstract":"Current deep networks are very data-hungry and benefit from training on largescale datasets, which are often time-consuming to collect and annotate. By contrast, synthetic data can be generated infinitely using generative models such as DALL-E and diffusion models, with minimal effort and cost. In this paper, we present DatasetDM, a generic dataset generation model that can produce diverse synthetic images and the corresponding high-quality perception annotations (e.g., segmentation masks, and depth). Our method builds upon the pre-trained diffusion model and extends text-guided image synthesis to perception data generation. We show that the rich latent code of the diffusion model can be effectively decoded as accurate perception annotations using a decoder module. Training the decoder only needs less than 1% (around 100 images) manually labeled images, enabling the generation of an infinitely large annotated dataset. Then these synthetic data can be used for training various perception models for downstream tasks. To showcase the power of the proposed approach, we generate datasets with rich dense pixel-wise labels for a wide range of downstream tasks, including semantic segmentation, instance segmentation, and depth estimation. Notably, it achieves 1) state-of-the-art results on semantic segmentation and instance segmentation; 2) significantly more robust on domain generalization than using the real data alone; and state-of-the-art results in zero-shot segmentation setting; and 3) flexibility for efficient application and novel task composition (e.g., image editing). The project website and code can be found at https://weijiawu.github.io/DatasetDM_page/ and https://github.com/showlab/DatasetDM, respectively","url_abs":"https://arxiv.org/abs/2308.06160v2","url_pdf":"https://arxiv.org/pdf/2308.06160v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"datasetdm-synthesizing-data-with-perception-1","repo_url":"https://github.com/showlab/datasetdm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"dataset-generation","task_name":"Dataset Generation"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"zero-shot-segmentation","task_name":"Zero Shot Segmentation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.06160","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.06160"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/showlab/datasetdm","reach":{"status":"ok"}}],"summary":{"ran":4,"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"a042902caede448b","entry":"diffusion_step","repo":"showlab/datasetdm","repo_kind":"official","path":"ptp_utils.py","file_url":"https://github.com/showlab/datasetdm/blob/HEAD/ptp_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a042902caede448b"}},{"code_sha256_prefix":"c41836f3e51fa8f1","entry":"gram_schmidt","repo":"showlab/datasetdm","repo_kind":"official","path":"model/depth_module.py","file_url":"https://github.com/showlab/datasetdm/blob/HEAD/model/depth_module.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c41836f3e51fa8f1"}},{"code_sha256_prefix":"0a5faf5529408221","entry":"is_tensor","repo":"showlab/datasetdm","repo_kind":"official","path":"model/outputs.py","file_url":"https://github.com/showlab/datasetdm/blob/HEAD/model/outputs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a5faf5529408221"}},{"code_sha256_prefix":"e3c3bb87bbe9b4a7","entry":"latent2image","repo":"showlab/datasetdm","repo_kind":"official","path":"ptp_utils.py","file_url":"https://github.com/showlab/datasetdm/blob/HEAD/ptp_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3c3bb87bbe9b4a7"}},{"code_sha256_prefix":"23d787190da4932e","entry":"resize_fn","repo":"showlab/datasetdm","repo_kind":"official","path":"model/depth_module.py","file_url":"https://github.com/showlab/datasetdm/blob/HEAD/model/depth_module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23d787190da4932e"}},{"code_sha256_prefix":"2f61baa7d4185642","entry":"text_under_image","repo":"showlab/datasetdm","repo_kind":"official","path":"ptp_utils.py","file_url":"https://github.com/showlab/datasetdm/blob/HEAD/ptp_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f61baa7d4185642"}},{"code_sha256_prefix":"d167441d997c52ce","entry":"get_matrix","repo":"showlab/datasetdm","repo_kind":"official","path":"seq_aligner.py","file_url":"https://github.com/showlab/datasetdm/blob/HEAD/seq_aligner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d167441d997c52ce"}},{"code_sha256_prefix":"81db21bf982ca0fe","entry":"get_matrix","repo":"showlab/datasetdm","repo_kind":"official","path":"seq_aligner.py","file_url":"https://github.com/showlab/datasetdm/blob/HEAD/seq_aligner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81db21bf982ca0fe"}},{"code_sha256_prefix":"beb8f6a30678a4d9","entry":"get_traceback_matrix","repo":"showlab/datasetdm","repo_kind":"official","path":"seq_aligner.py","file_url":"https://github.com/showlab/datasetdm/blob/HEAD/seq_aligner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"beb8f6a30678a4d9"}},{"code_sha256_prefix":"4015ab1f3ab9b881","entry":"proj","repo":"showlab/datasetdm","repo_kind":"official","path":"model/depth_module.py","file_url":"https://github.com/showlab/datasetdm/blob/HEAD/model/depth_module.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4015ab1f3ab9b881"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}