{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tencdm-understanding-the-properties-of","title":"TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings","arxiv_id":"2402.19097","date":"2024-02-29","proceeding":null,"authors":["Alexander Shabalin","Viacheslav Meshchaninov","Egor Chimbulatov","Vladislav Lapikov","Roman Kim","Grigory Bartosh","Dmitry Molchanov","Sergey Markov","Dmitry Vetrov"],"abstract":"This paper presents the Text Encoding Diffusion Model (TEncDM), a novel approach to diffusion modeling that operates in the space of pre-trained language model encodings. In contrast to traditionally used embeddings, encodings integrate contextual information. In our approach, we also employ a transformer-based decoder, specifically designed to incorporate context in the token prediction process. We conduct a comprehensive examination of the influence of the encoder, decoder, noise scheduler, and self-conditioning on zero-shot generation. Furthermore, we compare TEncDM with previous approaches on three conditional text generation tasks: QQP, XSum, and Wiki-Auto. The results show that TEncDM exhibits superior performance compared to existing non-autoregressive diffusion models. Our code is available at https://github.com/M0RJIQUE/tencdm.","url_abs":"https://arxiv.org/abs/2402.19097v3","url_pdf":"https://arxiv.org/pdf/2402.19097v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tencdm-understanding-the-properties-of","repo_url":"https://github.com/M0RJIQUE/tencdm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"conditional-text-generation","task_name":"Conditional Text Generation"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"qqp","task_name":"QQP"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.19097","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}