{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/any-order-gpt-as-masked-diffusion-model","title":"Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture","arxiv_id":"2506.19935","date":"2025-06-24","proceeding":null,"authors":["Shuchen Xue","Tianyu Xie","Tianyang Hu","Zijin Feng","Jiacheng Sun","Kenji Kawaguchi","Zhenguo Li","Zhi-Ming Ma"],"abstract":"Large language models (LLMs) predominantly use autoregressive (AR) approaches, but masked diffusion models (MDMs) are emerging as viable alternatives. A key challenge in comparing AR and MDM paradigms is their typical architectural difference: AR models are often decoder-only, while MDMs have largely been encoder-only. This practice of changing both the modeling paradigm and architecture simultaneously makes direct comparisons unfair, as it's hard to distinguish whether observed differences stem from the paradigm itself or the architectural shift. This research evaluates MDMs within a decoder-only framework to: (1) equitably compare MDM (as Any-Order AR, or AO-AR) and standard AR paradigms. Our investigation suggests that the standard AO-AR objective, which averages over all token permutations, may benefit from refinement, as many permutations appear less informative compared to the language's inherent left-to-right structure. (2) Investigate architectural influences (decoder-only vs. encoder-only) within MDMs. We demonstrate that while encoder-only MDMs model a simpler conditional probability space, decoder-only MDMs can achieve dramatic generation speedups ($\\sim25\\times$) and comparable perplexity with temperature annealing despite modeling a vastly larger space, highlighting key trade-offs. This work thus decouples core paradigm differences from architectural influences, offering insights for future model design. Code is available at https://github.com/scxue/AO-GPT-MDM.","url_abs":"https://arxiv.org/abs/2506.19935v1","url_pdf":"https://arxiv.org/pdf/2506.19935v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"any-order-gpt-as-masked-diffusion-model","repo_url":"https://github.com/scxue/ao-gpt-mdm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2506.19935","atlas_url":"https://app.syntology.ai/?focus=2506.19935","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.19935"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/scxue/ao-gpt-mdm","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/scxue/AO-GPT-MDM","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_honours":1,"unverified":6},"by_repo_kind":{"official":{"samples":14,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"2c1c6667a1ed50a0","entry":"AOGPT","repo":"scxue/AO-GPT-MDM","repo_kind":"official","path":"model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","file_url":"https://github.com/scxue/AO-GPT-MDM/blob/HEAD/model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c1c6667a1ed50a0"}},{"code_sha256_prefix":"311dd4150ecf301e","entry":"CausalSelfAttention","repo":"scxue/AO-GPT-MDM","repo_kind":"official","path":"model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","file_url":"https://github.com/scxue/AO-GPT-MDM/blob/HEAD/model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"311dd4150ecf301e"}},{"code_sha256_prefix":"f3bdc88f4e362aea","entry":"FinalLayer","repo":"scxue/AO-GPT-MDM","repo_kind":"official","path":"model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","file_url":"https://github.com/scxue/AO-GPT-MDM/blob/HEAD/model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3bdc88f4e362aea"}},{"code_sha256_prefix":"7d84ead0fe5b239f","entry":"RMSNorm","repo":"scxue/AO-GPT-MDM","repo_kind":"official","path":"model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","file_url":"https://github.com/scxue/AO-GPT-MDM/blob/HEAD/model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d84ead0fe5b239f"}},{"code_sha256_prefix":"576e1b9b9bcae8ff","entry":"lm1b_detokenizer","repo":"scxue/ao-gpt-mdm","repo_kind":"official","path":"data.py","file_url":"https://github.com/scxue/ao-gpt-mdm/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"576e1b9b9bcae8ff"}},{"code_sha256_prefix":"62fcb3912a967a50","entry":"modulate","repo":"scxue/ao-gpt-mdm","repo_kind":"official","path":"model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","file_url":"https://github.com/scxue/ao-gpt-mdm/blob/HEAD/model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"62fcb3912a967a50"}},{"code_sha256_prefix":"9f7fbdbe7ca280e0","entry":"ptb_detokenizer","repo":"scxue/ao-gpt-mdm","repo_kind":"official","path":"data.py","file_url":"https://github.com/scxue/ao-gpt-mdm/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f7fbdbe7ca280e0"}},{"code_sha256_prefix":"46c10627206a7795","entry":"wt_detokenizer","repo":"scxue/ao-gpt-mdm","repo_kind":"official","path":"data.py","file_url":"https://github.com/scxue/ao-gpt-mdm/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46c10627206a7795"}},{"code_sha256_prefix":"ad3472bf14169653","entry":"Block","repo":"scxue/AO-GPT-MDM","repo_kind":"official","path":"model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","file_url":"https://github.com/scxue/AO-GPT-MDM/blob/HEAD/model_AOGPT_AdaLN6_NoRep_cond_128_trunc_qknorm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad3472bf14169653"}},{"code_sha256_prefix":"d3ed17c7a19c2660","entry":"add_scale_fused","repo":"scxue/ao-gpt-mdm","repo_kind":"official","path":"model_AOGPT_generate.py","file_url":"https://github.com/scxue/ao-gpt-mdm/blob/HEAD/model_AOGPT_generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3ed17c7a19c2660"}},{"code_sha256_prefix":"a017bf60b761f427","entry":"gumbel_softmax","repo":"scxue/ao-gpt-mdm","repo_kind":"official","path":"catsample.py","file_url":"https://github.com/scxue/ao-gpt-mdm/blob/HEAD/catsample.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a017bf60b761f427"}},{"code_sha256_prefix":"ae815ba2af68a57f","entry":"modulate_fused","repo":"scxue/ao-gpt-mdm","repo_kind":"official","path":"model_AOGPT_generate.py","file_url":"https://github.com/scxue/ao-gpt-mdm/blob/HEAD/model_AOGPT_generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae815ba2af68a57f"}},{"code_sha256_prefix":"363f8d07ea5498f8","entry":"sample_categorical","repo":"scxue/ao-gpt-mdm","repo_kind":"official","path":"catsample.py","file_url":"https://github.com/scxue/ao-gpt-mdm/blob/HEAD/catsample.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"363f8d07ea5498f8"}},{"code_sha256_prefix":"7f1c917dfe447509","entry":"top_p_sampling_from_probs","repo":"scxue/ao-gpt-mdm","repo_kind":"official","path":"catsample.py","file_url":"https://github.com/scxue/ao-gpt-mdm/blob/HEAD/catsample.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f1c917dfe447509"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}