{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-evaluation-of-semantic-phenomena-in","title":"On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference","arxiv_id":"1804.09779","date":"2018-04-25","proceeding":"NAACL 2018 6","authors":["Adam Poliak","Yonatan Belinkov","James Glass","Benjamin Van Durme"],"abstract":"We propose a process for investigating the extent to which sentence\nrepresentations arising from neural machine translation (NMT) systems encode\ndistinct semantic phenomena. We use these representations as features to train\na natural language inference (NLI) classifier based on datasets recast from\nexisting semantic annotations. In applying this process to a representative NMT\nsystem, we find its encoder appears most suited to supporting inferences at the\nsyntax-semantics interface, as compared to anaphora resolution requiring\nworld-knowledge. We conclude with a discussion on the merits and potential\ndeficiencies of the existing process, and how it may be improved and extended\nas a broader framework for evaluating semantic coverage.","url_abs":"http://arxiv.org/abs/1804.09779v2","url_pdf":"http://arxiv.org/pdf/1804.09779v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-evaluation-of-semantic-phenomena-in","repo_url":"https://github.com/boknilev/nmt-repr-analysis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}