{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-22075","title":"Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison","arxiv_id":"2603.22075","date":"2026-03-23","proceeding":null,"authors":["Caio Vicentino"],"abstract":"We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput (~50K tokens/second), with MDLM requiring only 4.7% more wall-clock time. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity-fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct-n, lower Self-BLEU), at the cost of occasional grammatical inconsistencies. All code, trained checkpoints, and data pipelines are released for reproducibility.","url_abs":"https://arxiv.org/abs/2603.22075","url_pdf":"https://arxiv.org/pdf/2603.22075","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.22075","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.22075"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/caiovicentino/arche","reach":null}],"summary":{"unverified":9},"by_repo_kind":{"found_in_text":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"df002d2585b26d51","entry":"count_parameters","repo":"caiovicentino/arche","repo_kind":"found_in_text","path":"model.py","file_url":"https://github.com/caiovicentino/arche/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df002d2585b26d51"}},{"code_sha256_prefix":"89bc2d1f6eba841c","entry":"create_models","repo":"caiovicentino/arche","repo_kind":"found_in_text","path":"gpu/models.py","file_url":"https://github.com/caiovicentino/arche/blob/HEAD/gpu/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89bc2d1f6eba841c"}},{"code_sha256_prefix":"cb42e76e4ef89ca8","entry":"generate_ar_batch","repo":"caiovicentino/arche","repo_kind":"found_in_text","path":"eval_diversity.py","file_url":"https://github.com/caiovicentino/arche/blob/HEAD/eval_diversity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb42e76e4ef89ca8"}},{"code_sha256_prefix":"1adc464f6d3c4418","entry":"generate_mdlm_single","repo":"caiovicentino/arche","repo_kind":"found_in_text","path":"eval_diversity.py","file_url":"https://github.com/caiovicentino/arche/blob/HEAD/eval_diversity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1adc464f6d3c4418"}},{"code_sha256_prefix":"22d6fb48dd51a30d","entry":"load_data","repo":"caiovicentino/arche","repo_kind":"found_in_text","path":"gpu/arche_colab.py","file_url":"https://github.com/caiovicentino/arche/blob/HEAD/gpu/arche_colab.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22d6fb48dd51a30d"}},{"code_sha256_prefix":"520869c6f3ac5003","entry":"load_text","repo":"caiovicentino/arche","repo_kind":"found_in_text","path":"data.py","file_url":"https://github.com/caiovicentino/arche/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"520869c6f3ac5003"}},{"code_sha256_prefix":"3549fc9ad12536ac","entry":"prepare_data","repo":"caiovicentino/arche","repo_kind":"found_in_text","path":"data.py","file_url":"https://github.com/caiovicentino/arche/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3549fc9ad12536ac"}},{"code_sha256_prefix":"059f792a8b4f7d6e","entry":"tokenize_words","repo":"caiovicentino/arche","repo_kind":"found_in_text","path":"eval_diversity.py","file_url":"https://github.com/caiovicentino/arche/blob/HEAD/eval_diversity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"059f792a8b4f7d6e"}},{"code_sha256_prefix":"9c09903fa4847edc","entry":"train_one","repo":"caiovicentino/arche","repo_kind":"found_in_text","path":"gpu/arche_colab.py","file_url":"https://github.com/caiovicentino/arche/blob/HEAD/gpu/arche_colab.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c09903fa4847edc"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}