{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-complementarity-between-pre-training-1","title":"On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation","arxiv_id":"2209.03316","date":"2022-09-07","proceeding":"COLING 2022 10","authors":["Changtong Zan","Liang Ding","Li Shen","Yu Cao","Weifeng Liu","DaCheng Tao"],"abstract":"Pre-Training (PT) of text representations has been successfully applied to low-resource Neural Machine Translation (NMT). However, it usually fails to achieve notable gains (sometimes, even worse) on resource-rich NMT on par with its Random-Initialization (RI) counterpart. We take the first step to investigate the complementarity between PT and RI in resource-rich scenarios via two probing analyses, and find that: 1) PT improves NOT the accuracy, but the generalization by achieving flatter loss landscapes than that of RI; 2) PT improves NOT the confidence of lexical choice, but the negative diversity by assigning smoother lexical probability distributions than that of RI. Based on these insights, we propose to combine their complementarities with a model fusion algorithm that utilizes optimal transport to align neurons between PT and RI. Experiments on two resource-rich translation benchmarks, WMT'17 English-Chinese (20M) and WMT'19 English-German (36M), show that PT and RI could be nicely complementary to each other, achieving substantial improvements considering both translation accuracy, generalization, and negative diversity. Probing tools and code are released at: https://github.com/zanchangtong/PTvsRI.","url_abs":"https://arxiv.org/abs/2209.03316v3","url_pdf":"https://arxiv.org/pdf/2209.03316v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-complementarity-between-pre-training-1","repo_url":"https://github.com/zanchangtong/ptvsri","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"low-resource-neural-machine-translation-2","task_name":"Low Resource Neural Machine Translation"},{"task_slug":"low-resource-neural-machine-translation","task_name":"Low-Resource Neural Machine Translation"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.03316","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.03316"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zanchangtong/ptvsri","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"unverified":12},"by_repo_kind":{"official":{"samples":13,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5f4f4355be28c01e","entry":"isnan","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"exps/ground_metric.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/exps/ground_metric.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f4f4355be28c01e"}},{"code_sha256_prefix":"3432e05d0edb284c","entry":"Bucket_Sort","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"analysis/lpd.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/analysis/lpd.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3432e05d0edb284c"}},{"code_sha256_prefix":"6e072a02b03fd090","entry":"compute_entropy","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"analysis/lpd.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/analysis/lpd.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e072a02b03fd090"}},{"code_sha256_prefix":"06100713e91fedb0","entry":"get_activation_hook","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"exps/activation_utils.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/exps/activation_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06100713e91fedb0"}},{"code_sha256_prefix":"8317d614381abb14","entry":"get_backward_hook","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"exps/activation_utils.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/exps/activation_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8317d614381abb14"}},{"code_sha256_prefix":"f6f3a6dc7ddac11e","entry":"get_histogram","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"exps/utils.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/exps/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6f3a6dc7ddac11e"}},{"code_sha256_prefix":"95b7af714acdb480","entry":"ignore_bn","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"analysis/lls.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/analysis/lls.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95b7af714acdb480"}},{"code_sha256_prefix":"8c8e99db77825af5","entry":"normalize_filter","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"analysis/lls.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/analysis/lls.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c8e99db77825af5"}},{"code_sha256_prefix":"da3b5afa5a405deb","entry":"rstrip","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"exps/activation_utils.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/exps/activation_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da3b5afa5a405deb"}},{"code_sha256_prefix":"f50989b03391e732","entry":"state_2_fairseq_model","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"exps/convert_model.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/exps/convert_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f50989b03391e732"}},{"code_sha256_prefix":"6ec22691050b4140","entry":"word_blank","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"analysis/DuA/robust_add_noisy.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/analysis/DuA/robust_add_noisy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ec22691050b4140"}},{"code_sha256_prefix":"f00dcacce93a5ba4","entry":"word_dropout","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"analysis/DuA/robust_add_noisy.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/analysis/DuA/robust_add_noisy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f00dcacce93a5ba4"}},{"code_sha256_prefix":"9e5e05535c1d55a3","entry":"word_shuffle","repo":"zanchangtong/ptvsri","repo_kind":"official","path":"analysis/DuA/robust_add_noisy.py","file_url":"https://github.com/zanchangtong/ptvsri/blob/HEAD/analysis/DuA/robust_add_noisy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e5e05535c1d55a3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}