{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/finetuning-pretrained-transformers-into-rnns","title":"Finetuning Pretrained Transformers into RNNs","arxiv_id":"2103.13076","date":"2021-03-24","proceeding":"EMNLP 2021 11","authors":["Jungo Kasai","Hao Peng","Yizhe Zhang","Dani Yogatama","Gabriel Ilharco","Nikolaos Pappas","Yi Mao","Weizhu Chen","Noah A. Smith"],"abstract":"Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. But this comes with a significant computational cost, as the attention mechanism's complexity scales quadratically with sequence length. Efficient transformer variants have received increasing interest in recent works. Among them, a linear-complexity recurrent variant has proven well suited for autoregressive generation. It approximates the softmax attention with randomized or heuristic feature maps, but can be difficult to train and may yield suboptimal accuracy. This work aims to convert a pretrained transformer into its efficient recurrent counterpart, improving efficiency while maintaining accuracy. Specifically, we propose a swap-then-finetune procedure: in an off-the-shelf pretrained transformer, we replace the softmax attention with its linear-complexity recurrent alternative and then finetune. With a learned feature map, our approach provides an improved tradeoff between efficiency and accuracy over the standard transformer and other recurrent variants. We also show that the finetuning process has lower training cost relative to training these recurrent variants from scratch. As many models for natural language tasks are increasingly dependent on large-scale pretrained transformers, this work presents a viable approach to improving inference efficiency without repeating the expensive pretraining process.","url_abs":"https://arxiv.org/abs/2103.13076v2","url_pdf":"https://arxiv.org/pdf/2103.13076v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"finetuning-pretrained-transformers-into-rnns","repo_url":"https://github.com/hazyresearch/lolcats","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"finetuning-pretrained-transformers-into-rnns","repo_url":"https://github.com/yashbonde/RNN-sim","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"T2R + Pretrain","rank_in_archive_order":41,"of":89,"metrics":{"Test perplexity":"19.6","Validation perplexity":"19"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-french","task":"Machine Translation","dataset":"WMT2014 English-French","model":"T2R + Pretrain","rank_in_archive_order":19,"of":57,"metrics":{"BLEU score":"42.1"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"T2R + Pretrain","rank_in_archive_order":39,"of":91,"metrics":{"BLEU score":"28.7"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2017-chinese","task":"Machine Translation","dataset":"WMT2017 Chinese-English","model":"T2R + Pretrain","rank_in_archive_order":2,"of":3,"metrics":{"BLEU":"23.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.13076","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.13076"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazyresearch/lolcats","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yashbonde/RNN-sim","reach":null}],"summary":{"ran":2,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"listed":{"samples":8,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"c3f6efe6d4164aec","entry":"RNNAttention","repo":"yashbonde/RNN-sim","repo_kind":"listed","path":"t2rmodel.py","file_url":"https://github.com/yashbonde/RNN-sim/blob/HEAD/t2rmodel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c3f6efe6d4164aec"}},{"code_sha256_prefix":"512f9bd4885fc3fa","entry":"convert_attention","repo":"hazyresearch/lolcats","repo_kind":"listed","path":"src/model/convert_model.py","file_url":"https://github.com/hazyresearch/lolcats/blob/HEAD/src/model/convert_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"512f9bd4885fc3fa"}},{"code_sha256_prefix":"87cc0d3cf321bafc","entry":"traverse_layers","repo":"hazyresearch/lolcats","repo_kind":"listed","path":"src/model/convert_model.py","file_url":"https://github.com/hazyresearch/lolcats/blob/HEAD/src/model/convert_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"87cc0d3cf321bafc"}},{"code_sha256_prefix":"129bc8a38ada9fce","entry":"Block","repo":"yashbonde/RNN-sim","repo_kind":"listed","path":"t2rmodel.py","file_url":"https://github.com/yashbonde/RNN-sim/blob/HEAD/t2rmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"129bc8a38ada9fce"}},{"code_sha256_prefix":"b7d123852b18dc32","entry":"GenerationMixin","repo":"yashbonde/RNN-sim","repo_kind":"listed","path":"t2rmodel.py","file_url":"https://github.com/yashbonde/RNN-sim/blob/HEAD/t2rmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b7d123852b18dc32"}},{"code_sha256_prefix":"780a74bc5d6ff016","entry":"T2R","repo":"yashbonde/RNN-sim","repo_kind":"listed","path":"t2rmodel.py","file_url":"https://github.com/yashbonde/RNN-sim/blob/HEAD/t2rmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"780a74bc5d6ff016"}},{"code_sha256_prefix":"8057ebe8a31a2813","entry":"convert_llama_attention","repo":"hazyresearch/lolcats","repo_kind":"listed","path":"src/model/convert_model.py","file_url":"https://github.com/hazyresearch/lolcats/blob/HEAD/src/model/convert_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8057ebe8a31a2813"}},{"code_sha256_prefix":"7d2c368d797f9c8d","entry":"get_attention","repo":"hazyresearch/lolcats","repo_kind":"listed","path":"src/model/convert_model.py","file_url":"https://github.com/hazyresearch/lolcats/blob/HEAD/src/model/convert_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d2c368d797f9c8d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}