{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/it-s-easier-to-translate-out-of-english-than","title":"It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information","arxiv_id":"2005.02354","date":"2020-05-05","proceeding":"ACL 2020 6","authors":["Emanuele Bugliarello","Sabrina J. Mielke","Antonios Anastasopoulos","Ryan Cotterell","Naoaki Okazaki"],"abstract":"The performance of neural machine translation systems is commonly evaluated in terms of BLEU. However, due to its reliance on target language properties and generation, the BLEU metric does not allow an assessment of which translation directions are more difficult to model. In this paper, we propose cross-mutual information (XMI): an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. XMI allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty of the target-side generation component independent of the translation task. We then present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems. Code for replicating our experiments is available online at https://github.com/e-bug/nmt-difficulty.","url_abs":"https://arxiv.org/abs/2005.02354v2","url_pdf":"https://arxiv.org/pdf/2005.02354v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"it-s-easier-to-translate-out-of-english-than","repo_url":"https://github.com/e-bug/nmt-difficulty","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2005.02354","atlas_url":"https://app.syntology.ai/?focus=2005.02354","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.02354"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/e-bug/nmt-difficulty","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"90b79fe15491de83","entry":"is_master","repo":"e-bug/nmt-difficulty","repo_kind":"official","path":"fairseq-0.6.2/fairseq/distributed_utils.py","file_url":"https://github.com/e-bug/nmt-difficulty/blob/HEAD/fairseq-0.6.2/fairseq/distributed_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90b79fe15491de83"}},{"code_sha256_prefix":"82bdf940665aeeb4","entry":"safe_readline","repo":"e-bug/nmt-difficulty","repo_kind":"official","path":"fairseq-0.6.2/fairseq/binarizer.py","file_url":"https://github.com/e-bug/nmt-difficulty/blob/HEAD/fairseq-0.6.2/fairseq/binarizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82bdf940665aeeb4"}},{"code_sha256_prefix":"59755e76d7c3f48b","entry":"all_reduce","repo":"e-bug/nmt-difficulty","repo_kind":"official","path":"fairseq-0.6.2/fairseq/distributed_utils.py","file_url":"https://github.com/e-bug/nmt-difficulty/blob/HEAD/fairseq-0.6.2/fairseq/distributed_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59755e76d7c3f48b"}},{"code_sha256_prefix":"9eda7f80331d658c","entry":"dataset_dest_prefix","repo":"e-bug/nmt-difficulty","repo_kind":"official","path":"fairseq-0.6.2/preprocess.py","file_url":"https://github.com/e-bug/nmt-difficulty/blob/HEAD/fairseq-0.6.2/preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9eda7f80331d658c"}},{"code_sha256_prefix":"77c5626d767c9b7b","entry":"distributed_init","repo":"e-bug/nmt-difficulty","repo_kind":"official","path":"fairseq-0.6.2/fairseq/distributed_utils.py","file_url":"https://github.com/e-bug/nmt-difficulty/blob/HEAD/fairseq-0.6.2/fairseq/distributed_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77c5626d767c9b7b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}