{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/structured-denoising-diffusion-models-in","title":"Structured Denoising Diffusion Models in Discrete State-Spaces","arxiv_id":"2107.03006","date":"2021-07-07","proceeding":"NeurIPS 2021 12","authors":["Jacob Austin","Daniel D. Johnson","Jonathan Ho","Daniel Tarlow","Rianne van den Berg"],"abstract":"Denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) have shown impressive results on image and waveform generation in continuous state spaces. Here, we introduce Discrete Denoising Diffusion Probabilistic Models (D3PMs), diffusion-like generative models for discrete data that generalize the multinomial diffusion model of Hoogeboom et al. 2021, by going beyond corruption processes with uniform transition probabilities. This includes corruption with transition matrices that mimic Gaussian kernels in continuous space, matrices based on nearest neighbors in embedding space, and matrices that introduce absorbing states. The third allows us to draw a connection between diffusion models and autoregressive and mask-based generative models. We show that the choice of transition matrix is an important design decision that leads to improved results in image and text domains. We also introduce a new loss function that combines the variational lower bound with an auxiliary cross entropy loss. For text, this model class achieves strong results on character-level text generation while scaling to large vocabularies on LM1B. On the image dataset CIFAR-10, our models approach the sample quality and exceed the log-likelihood of the continuous-space DDPM model.","url_abs":"https://arxiv.org/abs/2107.03006v3","url_pdf":"https://arxiv.org/pdf/2107.03006v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"structured-denoising-diffusion-models-in","repo_url":"https://github.com/HKUNLP/Dream","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"structured-denoising-diffusion-models-in","repo_url":"https://github.com/hkunlp/reparam-discrete-diffusion","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"structured-denoising-diffusion-models-in","repo_url":"https://github.com/microsoft/evodiff","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"structured-denoising-diffusion-models-in","repo_url":"https://github.com/samb-t/unleashing-transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"structured-denoising-diffusion-models-in","repo_url":"https://github.com/xiaoiker/meta_dpm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"structured-denoising-diffusion-models-in","repo_url":"https://github.com/zhiminzhang0830/D3PM_Paddle","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"paddle","reach":{"status":"ok"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2107.03006","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.03006"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/samb-t/unleashing-transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xiaoiker/meta_dpm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HKUNLP/Dream","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hkunlp/reparam-discrete-diffusion","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/evodiff","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhiminzhang0830/D3PM_Paddle","reach":{"status":"ok"}}],"summary":{"ran":4,"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":5,"ran":5,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6a85860b6ec451b5","entry":"AbsorbingDiffusion","repo":"xiaoiker/meta_dpm","repo_kind":"listed","path":"models/absorbing_diffusion.py","file_url":"https://github.com/xiaoiker/meta_dpm/blob/HEAD/models/absorbing_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a85860b6ec451b5"}},{"code_sha256_prefix":"5d8c4051aef7ee8e","entry":"AbsorbingDiffusion","repo":"samb-t/unleashing-transformers","repo_kind":"listed","path":"models/absorbing_diffusion.py","file_url":"https://github.com/samb-t/unleashing-transformers/blob/HEAD/models/absorbing_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d8c4051aef7ee8e"}},{"code_sha256_prefix":"71964bb493b3fa09","entry":"Sampler","repo":"xiaoiker/meta_dpm","repo_kind":"listed","path":"models/absorbing_diffusion.py","file_url":"https://github.com/xiaoiker/meta_dpm/blob/HEAD/models/absorbing_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71964bb493b3fa09"}},{"code_sha256_prefix":"e7b664d9cbec19fd","entry":"Sampler","repo":"samb-t/unleashing-transformers","repo_kind":"listed","path":"models/absorbing_diffusion.py","file_url":"https://github.com/samb-t/unleashing-transformers/blob/HEAD/models/absorbing_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7b664d9cbec19fd"}},{"code_sha256_prefix":"e46586141f8fe8f5","entry":"q_sample","repo":"HKUNLP/Dream","repo_kind":"listed","path":"src/diffllm/gen_utils.py","file_url":"https://github.com/HKUNLP/Dream/blob/HEAD/src/diffllm/gen_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e46586141f8fe8f5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}