{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffusionbert-improving-generative-masked","title":"DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models","arxiv_id":"2211.15029","date":"2022-11-28","proceeding":null,"authors":["Zhengfu He","Tianxiang Sun","Kuanning Wang","Xuanjing Huang","Xipeng Qiu"],"abstract":"We present DiffusionBERT, a new generative masked language model based on discrete diffusion models. Diffusion models and many pre-trained language models have a shared training objective, i.e., denoising, making it possible to combine the two powerful models and enjoy the best of both worlds. On the one hand, diffusion models offer a promising training strategy that helps improve the generation quality. On the other hand, pre-trained denoising language models (e.g., BERT) can be used as a good initialization that accelerates convergence. We explore training BERT to learn the reverse process of a discrete diffusion process with an absorbing state and elucidate several designs to improve it. First, we propose a new noise schedule for the forward diffusion process that controls the degree of noise added at each step based on the information of each token. Second, we investigate several designs of incorporating the time step into BERT. Experiments on unconditional text generation demonstrate that DiffusionBERT achieves significant improvement over existing diffusion models for text (e.g., D3PM and Diffusion-LM) and previous generative masked language models in terms of perplexity and BLEU score.","url_abs":"https://arxiv.org/abs/2211.15029v2","url_pdf":"https://arxiv.org/pdf/2211.15029v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffusionbert-improving-generative-masked","repo_url":"https://github.com/hzfinfdu/diffusion-bert","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.15029","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.15029"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hzfinfdu/diffusion-bert","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":10},"by_repo_kind":{"official":{"samples":11,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"38f46ef4e03fee1c","entry":"create_position_ids_from_input_ids","repo":"hzfinfdu/diffusion-bert","repo_kind":"official","path":"models/modeling_roberta.py","file_url":"https://github.com/hzfinfdu/diffusion-bert/blob/HEAD/models/modeling_roberta.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38f46ef4e03fee1c"}},{"code_sha256_prefix":"6f15d047a8622d18","entry":"collate_fn","repo":"hzfinfdu/diffusion-bert","repo_kind":"official","path":"compute_elbo.py","file_url":"https://github.com/hzfinfdu/diffusion-bert/blob/HEAD/compute_elbo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f15d047a8622d18"}},{"code_sha256_prefix":"110e847251e7cce1","entry":"cross_entropy_with_probs","repo":"hzfinfdu/diffusion-bert","repo_kind":"official","path":"losses.py","file_url":"https://github.com/hzfinfdu/diffusion-bert/blob/HEAD/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"110e847251e7cce1"}},{"code_sha256_prefix":"841fce93f14f6675","entry":"dist1","repo":"hzfinfdu/diffusion-bert","repo_kind":"official","path":"compute_metric.py","file_url":"https://github.com/hzfinfdu/diffusion-bert/blob/HEAD/compute_metric.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"841fce93f14f6675"}},{"code_sha256_prefix":"35a119672af367f4","entry":"div4","repo":"hzfinfdu/diffusion-bert","repo_kind":"official","path":"compute_metric.py","file_url":"https://github.com/hzfinfdu/diffusion-bert/blob/HEAD/compute_metric.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"35a119672af367f4"}},{"code_sha256_prefix":"7b8175dc450ebd39","entry":"get_bleu","repo":"hzfinfdu/diffusion-bert","repo_kind":"official","path":"compute_metric.py","file_url":"https://github.com/hzfinfdu/diffusion-bert/blob/HEAD/compute_metric.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b8175dc450ebd39"}},{"code_sha256_prefix":"f93c2d73d355d807","entry":"get_timestep_embedding","repo":"hzfinfdu/diffusion-bert","repo_kind":"official","path":"models/modeling_bert_new_timestep.py","file_url":"https://github.com/hzfinfdu/diffusion-bert/blob/HEAD/models/modeling_bert_new_timestep.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f93c2d73d355d807"}},{"code_sha256_prefix":"24387783361281e8","entry":"kl_divergence_with_probs","repo":"hzfinfdu/diffusion-bert","repo_kind":"official","path":"losses.py","file_url":"https://github.com/hzfinfdu/diffusion-bert/blob/HEAD/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"24387783361281e8"}},{"code_sha256_prefix":"d009b630f4496e64","entry":"load_tf_weights_in_bert","repo":"hzfinfdu/diffusion-bert","repo_kind":"official","path":"models/modeling_bert.py","file_url":"https://github.com/hzfinfdu/diffusion-bert/blob/HEAD/models/modeling_bert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d009b630f4496e64"}},{"code_sha256_prefix":"1dd4ba483b72927b","entry":"p_forward","repo":"hzfinfdu/diffusion-bert","repo_kind":"official","path":"diffusion_condition.py","file_url":"https://github.com/hzfinfdu/diffusion-bert/blob/HEAD/diffusion_condition.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1dd4ba483b72927b"}},{"code_sha256_prefix":"59f7bb96a85f250f","entry":"process_fn_in_collate","repo":"hzfinfdu/diffusion-bert","repo_kind":"official","path":"compute_elbo.py","file_url":"https://github.com/hzfinfdu/diffusion-bert/blob/HEAD/compute_elbo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"59f7bb96a85f250f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}