{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/natural-language-generation-with-neural","title":"Natural Language Generation with Neural Variational Models","arxiv_id":"1808.09012","date":"2018-08-27","proceeding":null,"authors":["Hareesh Bahuleyan"],"abstract":"In this thesis, we explore the use of deep neural networks for generation of\nnatural language. Specifically, we implement two sequence-to-sequence neural\nvariational models - variational autoencoders (VAE) and variational\nencoder-decoders (VED). VAEs for text generation are difficult to train due to\nissues associated with the Kullback-Leibler (KL) divergence term of the loss\nfunction vanishing to zero. We successfully train VAEs by implementing\noptimization heuristics such as KL weight annealing and word dropout. We also\ndemonstrate the effectiveness of this continuous latent space through\nexperiments such as random sampling, linear interpolation and sampling from the\nneighborhood of the input. We argue that if VAEs are not designed\nappropriately, it may lead to bypassing connections which results in the latent\nspace being ignored during training. We show experimentally with the example of\ndecoder hidden state initialization that such bypassing connections degrade the\nVAE into a deterministic model, thereby reducing the diversity of generated\nsentences. We discover that the traditional attention mechanism used in\nsequence-to-sequence VED models serves as a bypassing connection, thereby\ndeteriorating the model's latent space. In order to circumvent this issue, we\npropose the variational attention mechanism where the attention context vector\nis modeled as a random variable that can be sampled from a distribution. We\nshow empirically using automatic evaluation metrics, namely entropy and\ndistinct measures, that our variational attention model generates more diverse\noutput sentences than the deterministic attention model. A qualitative analysis\nwith human evaluation study proves that our model simultaneously produces\nsentences that are of high quality and equally fluent as the ones generated by\nthe deterministic attention counterpart.","url_abs":"http://arxiv.org/abs/1808.09012v1","url_pdf":"http://arxiv.org/pdf/1808.09012v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"natural-language-generation-with-neural","repo_url":"https://github.com/HareeshBahuleyan/tf-var-attention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1808.09012","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}