{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/domain-robustness-in-neural-machine","title":"Domain Robustness in Neural Machine Translation","arxiv_id":"1911.03109","date":"2019-11-08","proceeding":"AMTA 2020 10","authors":["Mathias Müller","Annette Rios","Rico Sennrich"],"abstract":"Translating text that diverges from the training domain is a key challenge for machine translation. Domain robustness---the generalization of models to unseen test domains---is low for both statistical (SMT) and neural machine translation (NMT). In this paper, we study the performance of SMT and NMT models on out-of-domain test sets. We find that in unknown domains, SMT and NMT suffer from very different problems: SMT systems are mostly adequate but not fluent, while NMT systems are mostly fluent, but not adequate. For NMT, we identify such hallucinations (translations that are fluent but unrelated to the source) as a key reason for low domain robustness. To mitigate this problem, we empirically compare methods that are reported to improve adequacy or in-domain robustness in terms of their effectiveness at improving domain robustness. In experiments on German to English OPUS data, and German to Romansh (a low-resource setting) we find that several methods improve domain robustness. While those methods do lead to higher BLEU scores overall, they only slightly increase the adequacy of translations compared to SMT.","url_abs":"https://arxiv.org/abs/1911.03109v2","url_pdf":"https://arxiv.org/pdf/1911.03109v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"domain-robustness-in-neural-machine","repo_url":"https://github.com/ZurichNLP/domain-robustness","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"domain-robustness-in-neural-machine","repo_url":"https://github.com/ZurichNLP/sockeye","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"mxnet","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1911.03109","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.03109"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ZurichNLP/domain-robustness","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ZurichNLP/sockeye","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c368bc95cab7628c","entry":"average","repo":"ZurichNLP/sockeye","repo_kind":"official","path":"sockeye/average.py","file_url":"https://github.com/ZurichNLP/sockeye/blob/HEAD/sockeye/average.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c368bc95cab7628c"}},{"code_sha256_prefix":"51c1e44ee3d7ddbf","entry":"define_buckets","repo":"ZurichNLP/sockeye","repo_kind":"official","path":"sockeye/data_io.py","file_url":"https://github.com/ZurichNLP/sockeye/blob/HEAD/sockeye/data_io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51c1e44ee3d7ddbf"}},{"code_sha256_prefix":"e8b08e1f3ef35f68","entry":"define_parallel_buckets","repo":"ZurichNLP/sockeye","repo_kind":"official","path":"sockeye/data_io.py","file_url":"https://github.com/ZurichNLP/sockeye/blob/HEAD/sockeye/data_io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8b08e1f3ef35f68"}},{"code_sha256_prefix":"e0dddbd458aacc82","entry":"float_greater_or_equal","repo":"ZurichNLP/sockeye","repo_kind":"official","path":"sockeye/arguments.py","file_url":"https://github.com/ZurichNLP/sockeye/blob/HEAD/sockeye/arguments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0dddbd458aacc82"}},{"code_sha256_prefix":"6f9de756bd25f7f5","entry":"get_bucket","repo":"ZurichNLP/sockeye","repo_kind":"official","path":"sockeye/data_io.py","file_url":"https://github.com/ZurichNLP/sockeye/blob/HEAD/sockeye/data_io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f9de756bd25f7f5"}},{"code_sha256_prefix":"a1f7bd96e47f1862","entry":"int_greater_or_equal","repo":"ZurichNLP/sockeye","repo_kind":"official","path":"sockeye/arguments.py","file_url":"https://github.com/ZurichNLP/sockeye/blob/HEAD/sockeye/arguments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1f7bd96e47f1862"}},{"code_sha256_prefix":"1bcf34acc4bfc5b5","entry":"interleaved_matmul_encdec_qk","repo":"ZurichNLP/sockeye","repo_kind":"official","path":"sockeye/layers.py","file_url":"https://github.com/ZurichNLP/sockeye/blob/HEAD/sockeye/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1bcf34acc4bfc5b5"}},{"code_sha256_prefix":"7b06d9d749b67e62","entry":"interleaved_matmul_encdec_valatt","repo":"ZurichNLP/sockeye","repo_kind":"official","path":"sockeye/layers.py","file_url":"https://github.com/ZurichNLP/sockeye/blob/HEAD/sockeye/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b06d9d749b67e62"}},{"code_sha256_prefix":"dbfe68e8c251c012","entry":"load_args","repo":"ZurichNLP/sockeye","repo_kind":"official","path":"sockeye/arguments.py","file_url":"https://github.com/ZurichNLP/sockeye/blob/HEAD/sockeye/arguments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dbfe68e8c251c012"}},{"code_sha256_prefix":"0dd5254345198861","entry":"parallel_subsample","repo":"ZurichNLP/sockeye","repo_kind":"official","path":"sockeye/checkpoint_decoder.py","file_url":"https://github.com/ZurichNLP/sockeye/blob/HEAD/sockeye/checkpoint_decoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0dd5254345198861"}},{"code_sha256_prefix":"80f8f7625e2ab5ff","entry":"strategy_best","repo":"ZurichNLP/sockeye","repo_kind":"official","path":"sockeye/average.py","file_url":"https://github.com/ZurichNLP/sockeye/blob/HEAD/sockeye/average.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80f8f7625e2ab5ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}