{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/simple-fusion-return-of-the-language-model","title":"Simple Fusion: Return of the Language Model","arxiv_id":"1809.00125","date":"2018-09-01","proceeding":"WS 2018 10","authors":["Felix Stahlberg","James Cross","Veselin Stoyanov"],"abstract":"Neural Machine Translation (NMT) typically leverages monolingual data in\ntraining through backtranslation. We investigate an alternative simple method\nto use monolingual data for NMT training: We combine the scores of a\npre-trained and fixed language model (LM) with the scores of a translation\nmodel (TM) while the TM is trained from scratch. To achieve that, we train the\ntranslation model to predict the residual probability of the training data\nadded to the prediction of the LM. This enables the TM to focus its capacity on\nmodeling the source sentence since it can rely on the LM for fluency. We show\nthat our method outperforms previous approaches to integrate LMs into NMT while\nthe architecture is simpler as it does not require gating networks to balance\nTM and LM. We observe gains of between +0.24 and +2.36 BLEU on all four test\nsets (English-Turkish, Turkish-English, Estonian-English, Xhosa-English) on top\nof ensembles without LM. We compare our method with alternative ways to utilize\nmonolingual data such as backtranslation, shallow fusion, and cold fusion.","url_abs":"http://arxiv.org/abs/1809.00125v2","url_pdf":"http://arxiv.org/pdf/1809.00125v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"simple-fusion-return-of-the-language-model","repo_url":"https://github.com/fstahlberg/tensor2tensor-usr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1809.00125","atlas_url":"https://app.syntology.ai/?focus=1809.00125","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1809.00125"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fstahlberg/tensor2tensor-usr","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"unverified":9},"by_repo_kind":{"listed":{"samples":10,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"290febe7a42e8479","entry":"get_pairs","repo":"fstahlberg/tensor2tensor-usr","repo_kind":"listed","path":"usr/configs/problem_sn.py","file_url":"https://github.com/fstahlberg/tensor2tensor-usr/blob/HEAD/usr/configs/problem_sn.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"290febe7a42e8479"}},{"code_sha256_prefix":"d2b64e9c4cd1fd64","entry":"beta_pdf_unnorm_py","repo":"fstahlberg/tensor2tensor-usr","repo_kind":"listed","path":"usr/configs/problem_lt.py","file_url":"https://github.com/fstahlberg/tensor2tensor-usr/blob/HEAD/usr/configs/problem_lt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d2b64e9c4cd1fd64"}},{"code_sha256_prefix":"68965e673cdc8ea8","entry":"filter_by_ratio","repo":"fstahlberg/tensor2tensor-usr","repo_kind":"listed","path":"usr/configs/problem.py","file_url":"https://github.com/fstahlberg/tensor2tensor-usr/blob/HEAD/usr/configs/problem.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68965e673cdc8ea8"}},{"code_sha256_prefix":"7bea3e86dbb8c0c1","entry":"filter_by_ratio","repo":"fstahlberg/tensor2tensor-usr","repo_kind":"listed","path":"usr/configs/problem_lt.py","file_url":"https://github.com/fstahlberg/tensor2tensor-usr/blob/HEAD/usr/configs/problem_lt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7bea3e86dbb8c0c1"}},{"code_sha256_prefix":"73330fdc8f43ff5b","entry":"log_prob_from_logits","repo":"fstahlberg/tensor2tensor-usr","repo_kind":"listed","path":"usr/modalities/simplefusion.py","file_url":"https://github.com/fstahlberg/tensor2tensor-usr/blob/HEAD/usr/modalities/simplefusion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73330fdc8f43ff5b"}},{"code_sha256_prefix":"ca3dd3d2ae0b0ec7","entry":"norm_pdf","repo":"fstahlberg/tensor2tensor-usr","repo_kind":"listed","path":"usr/configs/problem.py","file_url":"https://github.com/fstahlberg/tensor2tensor-usr/blob/HEAD/usr/configs/problem.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca3dd3d2ae0b0ec7"}},{"code_sha256_prefix":"3640e8fdd2d5e5f3","entry":"print_data","repo":"fstahlberg/tensor2tensor-usr","repo_kind":"listed","path":"usr/utils.py","file_url":"https://github.com/fstahlberg/tensor2tensor-usr/blob/HEAD/usr/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3640e8fdd2d5e5f3"}},{"code_sha256_prefix":"9a38684bbe33a50b","entry":"print_shape","repo":"fstahlberg/tensor2tensor-usr","repo_kind":"listed","path":"usr/utils.py","file_url":"https://github.com/fstahlberg/tensor2tensor-usr/blob/HEAD/usr/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a38684bbe33a50b"}},{"code_sha256_prefix":"b66d4c7060cda11c","entry":"shift_left_3d","repo":"fstahlberg/tensor2tensor-usr","repo_kind":"listed","path":"usr/models/gibbs.py","file_url":"https://github.com/fstahlberg/tensor2tensor-usr/blob/HEAD/usr/models/gibbs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b66d4c7060cda11c"}},{"code_sha256_prefix":"7babd4b153a80cd8","entry":"smoothing_cross_entropy_seqls","repo":"fstahlberg/tensor2tensor-usr","repo_kind":"listed","path":"usr/modalities/seqls.py","file_url":"https://github.com/fstahlberg/tensor2tensor-usr/blob/HEAD/usr/modalities/seqls.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7babd4b153a80cd8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}