{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/discrete-autoencoders-for-sequence-models","title":"Discrete Autoencoders for Sequence Models","arxiv_id":"1801.09797","date":"2018-01-29","proceeding":"ICLR 2018 1","authors":["Łukasz Kaiser","Samy Bengio"],"abstract":"Recurrent models for sequences have been recently successful at many tasks,\nespecially for language modeling and machine translation. Nevertheless, it\nremains challenging to extract good representations from these models. For\ninstance, even though language has a clear hierarchical structure going from\ncharacters through words to sentences, it is not apparent in current language\nmodels. We propose to improve the representation in sequence models by\naugmenting current approaches with an autoencoder that is forced to compress\nthe sequence through an intermediate discrete latent space. In order to\npropagate gradients though this discrete representation we introduce an\nimproved semantic hashing technique. We show that this technique performs well\non a newly proposed quantitative efficiency measure. We also analyze latent\ncodes produced by the model showing how they correspond to words and phrases.\nFinally, we present an application of the autoencoder-augmented model to\ngenerating diverse translations.","url_abs":"http://arxiv.org/abs/1801.09797v1","url_pdf":"http://arxiv.org/pdf/1801.09797v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"discrete-autoencoders-for-sequence-models","repo_url":"https://github.com/tensorflow/tensor2tensor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"discrete-autoencoders-for-sequence-models","repo_url":"https://github.com/HB-Liu/Improved-SemHash-and-Gumbel-Softmax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1801.09797","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}