{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mtnt-a-testbed-for-machine-translation-of","title":"MTNT: A Testbed for Machine Translation of Noisy Text","arxiv_id":"1809.00388","date":"2018-09-02","proceeding":"EMNLP 2018 10","authors":["Paul Michel","Graham Neubig"],"abstract":"Noisy or non-standard input text can cause disastrous mistranslations in most\nmodern Machine Translation (MT) systems, and there has been growing research\ninterest in creating noise-robust MT systems. However, as of yet there are no\npublicly available parallel corpora of with naturally occurring noisy inputs\nand translations, and thus previous work has resorted to evaluating on\nsynthetically created datasets. In this paper, we propose a benchmark dataset\nfor Machine Translation of Noisy Text (MTNT), consisting of noisy comments on\nReddit (www.reddit.com) and professionally sourced translations. We\ncommissioned translations of English comments into French and Japanese, as well\nas French and Japanese comments into English, on the order of 7k-37k sentences\nper language pair. We qualitatively and quantitatively examine the types of\nnoise included in this dataset, then demonstrate that existing MT models fail\nbadly on a number of noise-related phenomena, even after performing adaptation\non a small training set of in-domain data. This indicates that this dataset can\nprovide an attractive testbed for methods tailored to handling noisy text in\nMT. The data is publicly available at www.cs.cmu.edu/~pmichel1/mtnt/.","url_abs":"http://arxiv.org/abs/1809.00388v1","url_pdf":"http://arxiv.org/pdf/1809.00388v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mtnt-a-testbed-for-machine-translation-of","repo_url":"https://github.com/pmichel31415/mtnt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"mtnt-a-testbed-for-machine-translation-of","repo_url":"https://github.com/MysteryVaibhav/robust_mtnt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[{"slug":"mtnt","name":"MTNT","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1809.00388","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1809.00388"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MysteryVaibhav/robust_mtnt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pmichel31415/mtnt","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1},"listed":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"db3f3f3a41b3121f","entry":"add_err_spl","repo":"MysteryVaibhav/robust_mtnt","repo_kind":"listed","path":"artificial_noise.py","file_url":"https://github.com/MysteryVaibhav/robust_mtnt/blob/HEAD/artificial_noise.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db3f3f3a41b3121f"}},{"code_sha256_prefix":"2ad8fe5ec76ada44","entry":"add_err_prof","repo":"MysteryVaibhav/robust_mtnt","repo_kind":"listed","path":"artificial_noise.py","file_url":"https://github.com/MysteryVaibhav/robust_mtnt/blob/HEAD/artificial_noise.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ad8fe5ec76ada44"}},{"code_sha256_prefix":"44ba17efd9a3446a","entry":"contains_url","repo":"pmichel31415/mtnt","repo_kind":"official","path":"src/text.py","file_url":"https://github.com/pmichel31415/mtnt/blob/HEAD/src/text.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44ba17efd9a3446a"}},{"code_sha256_prefix":"6b5e0c25095d38c5","entry":"load_wordlist","repo":"pmichel31415/mtnt","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/pmichel31415/mtnt/blob/HEAD/src/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b5e0c25095d38c5"}},{"code_sha256_prefix":"1d153eec847cd293","entry":"loadtxt","repo":"pmichel31415/mtnt","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/pmichel31415/mtnt/blob/HEAD/src/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d153eec847cd293"}},{"code_sha256_prefix":"618d8e55a8803212","entry":"normalize_punctuation","repo":"pmichel31415/mtnt","repo_kind":"official","path":"src/normalize_punctuation.py","file_url":"https://github.com/pmichel31415/mtnt/blob/HEAD/src/normalize_punctuation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"618d8e55a8803212"}},{"code_sha256_prefix":"4499c317153eeb2d","entry":"softmax","repo":"pmichel31415/mtnt","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/pmichel31415/mtnt/blob/HEAD/src/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4499c317153eeb2d"}},{"code_sha256_prefix":"f9e8dc330c08034f","entry":"strip_markdown","repo":"pmichel31415/mtnt","repo_kind":"official","path":"src/text.py","file_url":"https://github.com/pmichel31415/mtnt/blob/HEAD/src/text.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9e8dc330c08034f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}