{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/difformer-empowering-diffusion-model-on","title":"Empowering Diffusion Models on the Embedding Space for Text Generation","arxiv_id":"2212.09412","date":"2022-12-19","proceeding":null,"authors":["Zhujin Gao","Junliang Guo","Xu Tan","Yongxin Zhu","Fang Zhang","Jiang Bian","Linli Xu"],"abstract":"Diffusion models have achieved state-of-the-art synthesis quality on both visual and audio tasks, and recent works further adapt them to textual data by diffusing on the embedding space. In this paper, we conduct systematic studies of the optimization challenges encountered with both the embedding space and the denoising model, which have not been carefully explored. Firstly, the data distribution is learnable for embeddings, which may lead to the collapse of the embedding space and unstable training. To alleviate this problem, we propose a new objective called the anchor loss which is more efficient than previous methods. Secondly, we find the noise levels of conventional schedules are insufficient for training a desirable denoising model while introducing varying degrees of degeneration in consequence. To address this challenge, we propose a novel framework called noise rescaling. Based on the above analysis, we propose Difformer, an embedding diffusion model based on Transformer. Experiments on varieties of seminal text generation tasks show the effectiveness of the proposed methods and the superiority of Difformer over previous state-of-the-art embedding diffusion baselines.","url_abs":"https://arxiv.org/abs/2212.09412v3","url_pdf":"https://arxiv.org/pdf/2212.09412v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"difformer-empowering-diffusion-model-on","repo_url":"https://github.com/zhjgao/difformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-summarization","task_name":"Text Summarization"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.09412","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.09412"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhjgao/difformer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_honours":3,"ran":1,"unverified":5},"by_repo_kind":{"official":{"samples":11,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cfd76fd0d89574a4","entry":"approx_standard_normal_cdf","repo":"zhjgao/difformer","repo_kind":"official","path":"improved_diffusion/losses.py","file_url":"https://github.com/zhjgao/difformer/blob/HEAD/improved_diffusion/losses.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfd76fd0d89574a4"}},{"code_sha256_prefix":"2ab2316ac6fdd869","entry":"betas_for_alpha_bar","repo":"zhjgao/difformer","repo_kind":"official","path":"improved_diffusion/gaussian_diffusion.py","file_url":"https://github.com/zhjgao/difformer/blob/HEAD/improved_diffusion/gaussian_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ab2316ac6fdd869"}},{"code_sha256_prefix":"cd33283d615fb3d7","entry":"discretized_gaussian_log_likelihood","repo":"zhjgao/difformer","repo_kind":"official","path":"improved_diffusion/losses.py","file_url":"https://github.com/zhjgao/difformer/blob/HEAD/improved_diffusion/losses.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd33283d615fb3d7"}},{"code_sha256_prefix":"a086d6286a40b889","entry":"get_named_beta_schedule","repo":"zhjgao/difformer","repo_kind":"official","path":"improved_diffusion/gaussian_diffusion.py","file_url":"https://github.com/zhjgao/difformer/blob/HEAD/improved_diffusion/gaussian_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a086d6286a40b889"}},{"code_sha256_prefix":"bcd8b4acab199405","entry":"make_output_format","repo":"zhjgao/difformer","repo_kind":"official","path":"improved_diffusion/logger.py","file_url":"https://github.com/zhjgao/difformer/blob/HEAD/improved_diffusion/logger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bcd8b4acab199405"}},{"code_sha256_prefix":"cf2798b666b231ca","entry":"normal_kl","repo":"zhjgao/difformer","repo_kind":"official","path":"improved_diffusion/losses.py","file_url":"https://github.com/zhjgao/difformer/blob/HEAD/improved_diffusion/losses.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf2798b666b231ca"}},{"code_sha256_prefix":"45a64d31d754e5e9","entry":"build_ffn","repo":"zhjgao/difformer","repo_kind":"official","path":"difformer/utils.py","file_url":"https://github.com/zhjgao/difformer/blob/HEAD/difformer/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45a64d31d754e5e9"}},{"code_sha256_prefix":"a863803cdd5f3ce6","entry":"make_master_params","repo":"zhjgao/difformer","repo_kind":"official","path":"improved_diffusion/fp16_util.py","file_url":"https://github.com/zhjgao/difformer/blob/HEAD/improved_diffusion/fp16_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a863803cdd5f3ce6"}},{"code_sha256_prefix":"e515a67f7f32e76d","entry":"mpi_weighted_mean","repo":"zhjgao/difformer","repo_kind":"official","path":"improved_diffusion/logger.py","file_url":"https://github.com/zhjgao/difformer/blob/HEAD/improved_diffusion/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e515a67f7f32e76d"}},{"code_sha256_prefix":"0c6607473a4c4c55","entry":"profile","repo":"zhjgao/difformer","repo_kind":"official","path":"improved_diffusion/logger.py","file_url":"https://github.com/zhjgao/difformer/blob/HEAD/improved_diffusion/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c6607473a4c4c55"}},{"code_sha256_prefix":"30e43bcf12d042b0","entry":"unflatten_master_params","repo":"zhjgao/difformer","repo_kind":"official","path":"improved_diffusion/fp16_util.py","file_url":"https://github.com/zhjgao/difformer/blob/HEAD/improved_diffusion/fp16_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30e43bcf12d042b0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}