{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/token-level-adaptive-training-for-neural","title":"Token-level Adaptive Training for Neural Machine Translation","arxiv_id":"2010.04380","date":"2020-10-09","proceeding":"EMNLP 2020 11","authors":["Shuhao Gu","Jinchao Zhang","Fandong Meng","Yang Feng","Wanying Xie","Jie zhou","Dong Yu"],"abstract":"There exists a token imbalance phenomenon in natural language as different tokens appear with different frequencies, which leads to different learning difficulties for tokens in Neural Machine Translation (NMT). The vanilla NMT model usually adopts trivial equal-weighted objectives for target tokens with different frequencies and tends to generate more high-frequency tokens and less low-frequency tokens compared with the golden token distribution. However, low-frequency tokens may carry critical semantic information that will affect the translation quality once they are neglected. In this paper, we explored target token-level adaptive objectives based on token frequencies to assign appropriate weights for each target token during training. We aimed that those meaningful but relatively low-frequency words could be assigned with larger weights in objectives to encourage the model to pay more attention to these tokens. Our method yields consistent improvements in translation quality on ZH-EN, EN-RO, and EN-DE translation tasks, especially on sentences that contain more low-frequency tokens where we can get 1.68, 1.02, and 0.52 BLEU increases compared with baseline, respectively. Further analyses show that our method can also improve the lexical diversity of translation.","url_abs":"https://arxiv.org/abs/2010.04380v1","url_pdf":"https://arxiv.org/pdf/2010.04380v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"token-level-adaptive-training-for-neural","repo_url":"https://github.com/ictnlp/TLAT-NMT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.04380","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.04380"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ictnlp/TLAT-NMT","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"54945fa02c6cfa63","entry":"register_criterion","repo":"ictnlp/TLAT-NMT","repo_kind":"official","path":"fairseq/criterions/adaptive_loss.py","file_url":"https://github.com/ictnlp/TLAT-NMT/blob/HEAD/fairseq/criterions/adaptive_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54945fa02c6cfa63"}},{"code_sha256_prefix":"f648ca5af0b86257","entry":"AdaptiveLoss","repo":"ictnlp/TLAT-NMT","repo_kind":"official","path":"fairseq/criterions/adaptive_loss.py","file_url":"https://github.com/ictnlp/TLAT-NMT/blob/HEAD/fairseq/criterions/adaptive_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f648ca5af0b86257"}},{"code_sha256_prefix":"448f8b7441b9ccd5","entry":"FairseqCriterion","repo":"ictnlp/TLAT-NMT","repo_kind":"official","path":"fairseq/criterions/adaptive_loss.py","file_url":"https://github.com/ictnlp/TLAT-NMT/blob/HEAD/fairseq/criterions/adaptive_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"448f8b7441b9ccd5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}