{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-long-tailed-phenomena-in-neural-machine","title":"On Long-Tailed Phenomena in Neural Machine Translation","arxiv_id":"2010.04924","date":"2020-10-10","proceeding":"Findings of the Association for Computational Linguistics 2020","authors":["Vikas Raunak","Siddharth Dalmia","Vivek Gupta","Florian Metze"],"abstract":"State-of-the-art Neural Machine Translation (NMT) models struggle with generating low-frequency tokens, tackling which remains a major challenge. The analysis of long-tailed phenomena in the context of structured prediction tasks is further hindered by the added complexities of search during inference. In this work, we quantitatively characterize such long-tailed phenomena at two levels of abstraction, namely, token classification and sequence generation. We propose a new loss function, the Anti-Focal loss, to better adapt model training to the structural dependencies of conditional text generation by incorporating the inductive biases of beam search in the training process. We show the efficacy of the proposed technique on a number of Machine Translation (MT) datasets, demonstrating that it leads to significant gains over cross-entropy across different language pairs, especially on the generation of low-frequency words. We have released the code to reproduce our results.","url_abs":"https://arxiv.org/abs/2010.04924v1","url_pdf":"https://arxiv.org/pdf/2010.04924v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-long-tailed-phenomena-in-neural-machine","repo_url":"https://github.com/vyraun/long-tailed","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"conditional-text-generation","task_name":"Conditional Text Generation"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"structured-prediction","task_name":"Structured Prediction"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"token-classification","task_name":"Token Classification"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"token-classification","task_name":"token-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2010.04924","atlas_url":"https://app.syntology.ai/?focus=2010.04924","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.04924"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/vyraun/long-tailed","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"170e2b9970e01fa2","entry":"get_idf_score","repo":"vyraun/long-tailed","repo_kind":"official","path":"analysis/normalize_last_layer.py","file_url":"https://github.com/vyraun/long-tailed/blob/HEAD/analysis/normalize_last_layer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"170e2b9970e01fa2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}