{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/context-dependent-word-representation-for","title":"Context-Dependent Word Representation for Neural Machine Translation","arxiv_id":"1607.00578","date":"2016-07-03","proceeding":null,"authors":["Heeyoul Choi","Kyunghyun Cho","Yoshua Bengio"],"abstract":"We first observe a potential weakness of continuous vector representations of\nsymbols in neural machine translation. That is, the continuous vector\nrepresentation, or a word embedding vector, of a symbol encodes multiple\ndimensions of similarity, equivalent to encoding more than one meaning of the\nword. This has the consequence that the encoder and decoder recurrent networks\nin neural machine translation need to spend substantial amount of their\ncapacity in disambiguating source and target words based on the context which\nis defined by a source sentence. Based on this observation, in this paper we\npropose to contextualize the word embedding vectors using a nonlinear\nbag-of-words representation of the source sentence. Additionally, we propose to\nrepresent special tokens (such as numbers, proper nouns and acronyms) with\ntyped symbols to facilitate translating those words that are not well-suited to\nbe translated via continuous vectors. The experiments on En-Fr and En-De reveal\nthat the proposed approaches of contextualization and symbolization improves\nthe translation quality of neural machine translation systems significantly.","url_abs":"http://arxiv.org/abs/1607.00578v1","url_pdf":"http://arxiv.org/pdf/1607.00578v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"context-dependent-word-representation-for","repo_url":"https://github.com/kyunghyuncho/WordVectorManifold","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1607.00578","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}