{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fine-tuning-by-curriculum-learning-for-non","title":"Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation","arxiv_id":"1911.08717","date":"2019-11-20","proceeding":null,"authors":["Junliang Guo","Xu Tan","Linli Xu","Tao Qin","Enhong Chen","Tie-Yan Liu"],"abstract":"Non-autoregressive translation (NAT) models remove the dependence on previous target tokens and generate all target tokens in parallel, resulting in significant inference speedup but at the cost of inferior translation accuracy compared to autoregressive translation (AT) models. Considering that AT models have higher accuracy and are easier to train than NAT models, and both of them share the same model configurations, a natural idea to improve the accuracy of NAT models is to transfer a well-trained AT model to an NAT model through fine-tuning. However, since AT and NAT models differ greatly in training strategy, straightforward fine-tuning does not work well. In this work, we introduce curriculum learning into fine-tuning for NAT. Specifically, we design a curriculum in the fine-tuning process to progressively switch the training from autoregressive generation to non-autoregressive generation. Experiments on four benchmark translation datasets show that the proposed method achieves good improvement (more than $1$ BLEU score) over previous NAT baselines in terms of translation accuracy, and greatly speed up (more than $10$ times) the inference process over AT baselines.","url_abs":"https://arxiv.org/abs/1911.08717v2","url_pdf":"https://arxiv.org/pdf/1911.08717v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fine-tuning-by-curriculum-learning-for-non","repo_url":"https://github.com/lemmonation/fcl-nat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"fine-tuning-by-curriculum-learning-for-non","repo_url":"https://github.com/josephch405/curriculum-nmt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1911.08717","atlas_url":"https://app.syntology.ai/?focus=1911.08717","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.08717"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lemmonation/fcl-nat","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/josephch405/curriculum-nmt","reach":null}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"44e847faaba1328c","entry":"add_standard_attention_hparams","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/layers/common_attention.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/layers/common_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44e847faaba1328c"}},{"code_sha256_prefix":"1cca0cf169d3512e","entry":"attention_bias_without_self","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/models/transformer_nat_cl_word.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/models/transformer_nat_cl_word.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1cca0cf169d3512e"}},{"code_sha256_prefix":"ff01c7e565eb434e","entry":"copy_batches","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/models/transformer_nat_cl_word.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/models/transformer_nat_cl_word.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff01c7e565eb434e"}},{"code_sha256_prefix":"62078081af34b8ee","entry":"hard_sigmoid","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/layers/common_layers.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/layers/common_layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62078081af34b8ee"}},{"code_sha256_prefix":"dfdf8307a155fa8b","entry":"hard_tanh","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/layers/common_layers.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/layers/common_layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfdf8307a155fa8b"}},{"code_sha256_prefix":"7e2bc85a4d4189a9","entry":"is_in_expr","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/data_generators/algorithmic_math.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/data_generators/algorithmic_math.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e2bc85a4d4189a9"}},{"code_sha256_prefix":"b1896a2536602d28","entry":"lower_endian_to_number","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/data_generators/algorithmic.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/data_generators/algorithmic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1896a2536602d28"}},{"code_sha256_prefix":"916cc023d1d1ddbd","entry":"random_expr","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/data_generators/algorithmic_math.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/data_generators/algorithmic_math.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"916cc023d1d1ddbd"}},{"code_sha256_prefix":"0a76872f8ddc4b1e","entry":"random_expr_with_required_var","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/data_generators/algorithmic_math.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/data_generators/algorithmic_math.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a76872f8ddc4b1e"}},{"code_sha256_prefix":"c53fad116e23fd5d","entry":"saturating_sigmoid","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/layers/common_layers.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/layers/common_layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c53fad116e23fd5d"}},{"code_sha256_prefix":"68fe9a15fc61f6bd","entry":"zipf_distribution","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/data_generators/algorithmic.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/data_generators/algorithmic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68fe9a15fc61f6bd"}},{"code_sha256_prefix":"d0d6f2cc06f67920","entry":"zipf_random_sample","repo":"lemmonation/fcl-nat","repo_kind":"official","path":"tensor2tensor/data_generators/algorithmic.py","file_url":"https://github.com/lemmonation/fcl-nat/blob/HEAD/tensor2tensor/data_generators/algorithmic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0d6f2cc06f67920"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}