{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/your-absorbing-discrete-diffusion-secretly","title":"Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data","arxiv_id":"2406.03736","date":"2024-06-06","proceeding":null,"authors":["Jingyang Ou","Shen Nie","Kaiwen Xue","Fengqi Zhu","Jiacheng Sun","Zhenguo Li","Chongxuan Li"],"abstract":"Discrete diffusion models with absorbing processes have shown promise in language modeling. The key quantities to be estimated are the ratios between the marginal probabilities of two transitive states at all timesteps, called the concrete score. In this paper, we reveal that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in an analytic form. Motivated by this finding, we propose reparameterized absorbing discrete diffusion (RADD), a dedicated diffusion model without time-condition that characterizes the time-independent conditional probabilities. Besides its simplicity, RADD can reduce the number of function evaluations (NFEs) by caching the output of the time-independent network when the noisy sample remains unchanged in a sampling interval. Empirically, RADD is up to 3.5 times faster while achieving similar performance with the strongest baseline. Built upon the new perspective of conditional distributions, we further unify absorbing discrete diffusion and any-order autoregressive models (AO-ARMs), showing that the upper bound on the negative log-likelihood for the diffusion model can be interpreted as an expected negative log-likelihood for AO-ARMs. Further, our RADD models achieve SOTA performance among diffusion models on 5 zero-shot language modeling benchmarks (measured by perplexity) at the GPT-2 scale. Our code is available at https://github.com/ML-GSAI/RADD.","url_abs":"https://arxiv.org/abs/2406.03736v2","url_pdf":"https://arxiv.org/pdf/2406.03736v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"your-absorbing-discrete-diffusion-secretly","repo_url":"https://github.com/ml-gsai/radd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"your-absorbing-discrete-diffusion-secretly","repo_url":"https://github.com/liusulin/ddpd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-2","method_name":"GPT-2"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.03736","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.03736"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liusulin/ddpd","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-gsai/radd","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"ran_honours":1,"ran_fixture":1,"unverified":8},"by_repo_kind":{"official":{"samples":13,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"576e1b9b9bcae8ff","entry":"lm1b_detokenizer","repo":"ml-gsai/radd","repo_kind":"official","path":"data.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"576e1b9b9bcae8ff"}},{"code_sha256_prefix":"9f7fbdbe7ca280e0","entry":"ptb_detokenizer","repo":"ml-gsai/radd","repo_kind":"official","path":"data.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f7fbdbe7ca280e0"}},{"code_sha256_prefix":"8526ce179df04bd4","entry":"rotate_half","repo":"ml-gsai/radd","repo_kind":"official","path":"model/rotary.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/model/rotary.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8526ce179df04bd4"}},{"code_sha256_prefix":"172aa319a831b3ab","entry":"sample_categorical","repo":"ml-gsai/radd","repo_kind":"official","path":"catsample.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/catsample.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"172aa319a831b3ab"}},{"code_sha256_prefix":"46c10627206a7795","entry":"wt_detokenizer","repo":"ml-gsai/radd","repo_kind":"official","path":"data.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46c10627206a7795"}},{"code_sha256_prefix":"7df53e0f36bfadb4","entry":"Batch_Uniform_Sampler","repo":"ml-gsai/radd","repo_kind":"official","path":"losses.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7df53e0f36bfadb4"}},{"code_sha256_prefix":"866bafb81f8da612","entry":"add_noise_lambda","repo":"ml-gsai/radd","repo_kind":"official","path":"noise_lib.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/noise_lib.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"866bafb81f8da612"}},{"code_sha256_prefix":"5dc01eaf879d151a","entry":"add_noise_t","repo":"ml-gsai/radd","repo_kind":"official","path":"noise_lib.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/noise_lib.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5dc01eaf879d151a"}},{"code_sha256_prefix":"b3c9c4aebf03fa97","entry":"apply_rotary_pos_emb","repo":"ml-gsai/radd","repo_kind":"official","path":"model/rotary.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/model/rotary.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b3c9c4aebf03fa97"}},{"code_sha256_prefix":"92369c72d11b7baf","entry":"direct_sampling","repo":"ml-gsai/radd","repo_kind":"official","path":"catsample.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/catsample.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"92369c72d11b7baf"}},{"code_sha256_prefix":"c7a7325b30d3491b","entry":"get_noise","repo":"ml-gsai/radd","repo_kind":"official","path":"noise_lib.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/noise_lib.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7a7325b30d3491b"}},{"code_sha256_prefix":"32fa27b9da9341dc","entry":"get_optimizer","repo":"ml-gsai/radd","repo_kind":"official","path":"losses.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32fa27b9da9341dc"}},{"code_sha256_prefix":"a017bf60b761f427","entry":"gumbel_softmax","repo":"ml-gsai/radd","repo_kind":"official","path":"catsample.py","file_url":"https://github.com/ml-gsai/radd/blob/HEAD/catsample.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a017bf60b761f427"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}