{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-layers-of-representation-in-neural","title":"Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks","arxiv_id":"1801.07772","date":"2018-01-23","proceeding":"IJCNLP 2017 11","authors":["Yonatan Belinkov","Lluís Màrquez","Hassan Sajjad","Nadir Durrani","Fahim Dalvi","James Glass"],"abstract":"While neural machine translation (NMT) models provide improved translation\nquality in an elegant, end-to-end framework, it is less clear what they learn\nabout language. Recent work has started evaluating the quality of vector\nrepresentations learned by NMT models on morphological and syntactic tasks. In\nthis paper, we investigate the representations learned at different layers of\nNMT encoders. We train NMT systems on parallel data and use the trained models\nto extract features for training a classifier on two tasks: part-of-speech and\nsemantic tagging. We then measure the performance of the classifier as a proxy\nto the quality of the original NMT model for the given task. Our quantitative\nanalysis yields interesting insights regarding representation learning in NMT\nmodels. For instance, we find that higher layers are better at learning\nsemantics while lower layers tend to be better for part-of-speech tagging. We\nalso observe little effect of the target language on source-side\nrepresentations, especially with higher quality NMT models.","url_abs":"http://arxiv.org/abs/1801.07772v1","url_pdf":"http://arxiv.org/pdf/1801.07772v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-layers-of-representation-in-neural","repo_url":"https://github.com/boknilev/nmt-repr-analysis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"part-of-speech-tagging","task_name":"Part-Of-Speech Tagging"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1801.07772","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}