{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/remasking-discrete-diffusion-models-with","title":"Remasking Discrete Diffusion Models with Inference-Time Scaling","arxiv_id":"2503.00307","date":"2025-03-01","proceeding":null,"authors":["Guanghan Wang","Yair Schiff","Subham Sekhar Sahoo","Volodymyr Kuleshov"],"abstract":"Part of the success of diffusion models stems from their ability to perform iterative refinement, i.e., repeatedly correcting outputs during generation. However, modern masked discrete diffusion lacks this capability: when a token is generated, it cannot be updated again, even when it introduces an error. Here, we address this limitation by introducing the remasking diffusion model (ReMDM) sampler, a method that can be applied to pretrained masked diffusion models in a principled way and that is derived from a discrete diffusion model with a custom remasking backward process. Most interestingly, ReMDM endows discrete diffusion with a form of inference-time compute scaling. By increasing the number of sampling steps, ReMDM generates natural language outputs that approach the quality of autoregressive models, whereas when the computation budget is limited, ReMDM better maintains quality. ReMDM also improves sample quality of masked diffusion models for discretized images, and in scientific domains such as molecule design, ReMDM facilitates diffusion guidance and pushes the Pareto frontier of controllability relative to classical masking and uniform noise diffusion. We provide the code along with a blog post on the project page: https://remdm.github.io.","url_abs":"https://arxiv.org/abs/2503.00307v1","url_pdf":"https://arxiv.org/pdf/2503.00307v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"remasking-discrete-diffusion-models-with","repo_url":"https://github.com/kuleshov-group/remdm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.00307","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.00307"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kuleshov-group/remdm","reach":null}],"summary":{"ran":10,"ran_draft_wrong":2,"ran_fixture":1,"unverified":4},"by_repo_kind":{"listed":{"samples":17,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8a96d6ee76d899fb","entry":"BPD","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a96d6ee76d899fb"}},{"code_sha256_prefix":"2252e72b7e6ff199","entry":"CosineNoise","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2252e72b7e6ff199"}},{"code_sha256_prefix":"f3c78ccd48148489","entry":"CosineSqrNoise","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f3c78ccd48148489"}},{"code_sha256_prefix":"a8eb0d121c7ecbcc","entry":"FaultTolerantDistributedSampler","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a8eb0d121c7ecbcc"}},{"code_sha256_prefix":"06f92d50089269a3","entry":"GeometricNoise","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06f92d50089269a3"}},{"code_sha256_prefix":"23091ee635899def","entry":"Linear","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"23091ee635899def"}},{"code_sha256_prefix":"721fda60a2762576","entry":"LogLinearNoise","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"721fda60a2762576"}},{"code_sha256_prefix":"f0c0a2810b675abb","entry":"Loss","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0c0a2810b675abb"}},{"code_sha256_prefix":"e3f789992ecfe962","entry":"Perplexity","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e3f789992ecfe962"}},{"code_sha256_prefix":"ca44ec1d6c02bb7d","entry":"RandomFaultTolerantSampler","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca44ec1d6c02bb7d"}},{"code_sha256_prefix":"1c47525224ba228c","entry":"_sample_categorical","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1c47525224ba228c"}},{"code_sha256_prefix":"30c96381b7f09da5","entry":"_unsqueeze","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30c96381b7f09da5"}},{"code_sha256_prefix":"2174721db8d9cdb6","entry":"get_noise","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2174721db8d9cdb6"}},{"code_sha256_prefix":"2c650372d2d0b176","entry":"Diffusion","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c650372d2d0b176"}},{"code_sha256_prefix":"c7461f97befe2293","entry":"NLL","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7461f97befe2293"}},{"code_sha256_prefix":"4360fa80bd3c0ebd","entry":"Noise","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4360fa80bd3c0ebd"}},{"code_sha256_prefix":"1665eba64e29d70c","entry":"print_nans","repo":"kuleshov-group/remdm","repo_kind":"listed","path":"diffusion.py","file_url":"https://github.com/kuleshov-group/remdm/blob/HEAD/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1665eba64e29d70c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}