{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improved-variational-autoencoders-for-text","title":"Improved Variational Autoencoders for Text Modeling using Dilated Convolutions","arxiv_id":"1702.08139","date":"2017-02-27","proceeding":"ICML 2017 8","authors":["Zichao Yang","Zhiting Hu","Ruslan Salakhutdinov","Taylor Berg-Kirkpatrick"],"abstract":"Recent work on generative modeling of text has found that variational\nauto-encoders (VAE) incorporating LSTM decoders perform worse than simpler LSTM\nlanguage models (Bowman et al., 2015). This negative result is so far poorly\nunderstood, but has been attributed to the propensity of LSTM decoders to\nignore conditioning information from the encoder. In this paper, we experiment\nwith a new type of decoder for VAE: a dilated CNN. By changing the decoder's\ndilation architecture, we control the effective context from previously\ngenerated words. In experiments, we find that there is a trade off between the\ncontextual capacity of the decoder and the amount of encoding information used.\nWe show that with the right decoder, VAE can outperform LSTM language models.\nWe demonstrate perplexity gains on two datasets, representing the first\npositive experimental result on the use VAE for generative modeling of text.\nFurther, we conduct an in-depth investigation of the use of VAE (with our new\ndecoding architecture) for semi-supervised and unsupervised labeling tasks,\ndemonstrating gains over several strong baselines.","url_abs":"http://arxiv.org/abs/1702.08139v2","url_pdf":"http://arxiv.org/pdf/1702.08139v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improved-variational-autoencoders-for-text","repo_url":"https://github.com/harvardnlp/sa-vae","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"improved-variational-autoencoders-for-text","repo_url":"https://github.com/kefirski/contiguous-succotash","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"improved-variational-autoencoders-for-text","repo_url":"https://github.com/ryokamoi/dcnn_textvae","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/text-generation-on-yahoo-questions","task":"Text Generation","dataset":"Yahoo Questions","model":"CNN-VAE","rank_in_archive_order":3,"of":3,"metrics":{"KL":"10.0","NLL":"332.1","Perplexity":"63.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1702.08139","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}