{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/with-greater-text-comes-greater-necessity","title":"With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation","arxiv_id":"2401.11504","date":"2024-01-21","proceeding":null,"authors":["Y. Wang","D. Ma","D. Cai"],"abstract":"Long text generation, such as novel writing and discourse-level translation with extremely long contexts, presents significant challenges to current language models. Existing methods mainly focus on extending the model's context window through strategies like length extrapolation. However, these approaches demand substantial hardware resources during the training and/or inference phases. Our proposed method, Temp-Lora, introduces an alternative concept. Instead of relying on the KV cache to store all context information, we embeds this information directly into a temporary Lora module. In the process of long text generation, this module is progressively trained with text generated previously. This approach not only efficiently preserves contextual knowledge but also prevents any permanent alteration to the model's parameters given that the module is discarded post-generation. Extensive experiments on the PG19 language modeling benchmark and the GuoFeng discourse-level translation benchmark validate the effectiveness of Temp-Lora. Our results show that: 1) Temp-Lora substantially enhances generation quality for long text, as indicated by a 13.2% decrease in perplexity (PPL) on a subset of PG19, and a 29.3% decrease in PPL along with a 113.2% increase in BLEU score on a subset of GuoFeng, 2) Temp-Lora is compatible with and enhances most existing long text generation methods, and 3) Temp-Lora can greatly reduce computational costs by shortening the context window. For example, we can ensure a moderate improvement in generation quality (a decrease of 3.8% in PPL) while enabling a 51.5% memory usage reduction and a 60.0% decrease in latency for inference.","url_abs":"https://arxiv.org/abs/2401.11504v3","url_pdf":"https://arxiv.org/pdf/2401.11504v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"with-greater-text-comes-greater-necessity","repo_url":"https://github.com/temporarylora/temp-lora","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.11504","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.11504"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/temporarylora/temp-lora","reach":{"status":"ok"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"c0a42c97116fb6ca","entry":"get_zh_text_from_prompt","repo":"temporarylora/temp-lora","repo_kind":"official","path":"eval_results/guo_feng.py","file_url":"https://github.com/temporarylora/temp-lora/blob/HEAD/eval_results/guo_feng.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0a42c97116fb6ca"}},{"code_sha256_prefix":"e7e08656f89e64cd","entry":"inference_once","repo":"temporarylora/temp-lora","repo_kind":"official","path":"trainer/acc_pg19_trainer.py","file_url":"https://github.com/temporarylora/temp-lora/blob/HEAD/trainer/acc_pg19_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7e08656f89e64cd"}},{"code_sha256_prefix":"581873e53ab0fe30","entry":"json_stringify","repo":"temporarylora/temp-lora","repo_kind":"official","path":"helper.py","file_url":"https://github.com/temporarylora/temp-lora/blob/HEAD/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"581873e53ab0fe30"}},{"code_sha256_prefix":"28dee895e38130b6","entry":"load_json","repo":"temporarylora/temp-lora","repo_kind":"official","path":"helper.py","file_url":"https://github.com/temporarylora/temp-lora/blob/HEAD/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28dee895e38130b6"}},{"code_sha256_prefix":"d9b3ca921e12dd64","entry":"load_jsonline","repo":"temporarylora/temp-lora","repo_kind":"official","path":"helper.py","file_url":"https://github.com/temporarylora/temp-lora/blob/HEAD/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9b3ca921e12dd64"}},{"code_sha256_prefix":"6266b1ace9d09be0","entry":"to_device","repo":"temporarylora/temp-lora","repo_kind":"official","path":"trainer/acc_guo_feng_trainer.py","file_url":"https://github.com/temporarylora/temp-lora/blob/HEAD/trainer/acc_guo_feng_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6266b1ace9d09be0"}},{"code_sha256_prefix":"70a4a56320ca2468","entry":"to_device","repo":"temporarylora/temp-lora","repo_kind":"official","path":"trainer/acc_pg19_trainer.py","file_url":"https://github.com/temporarylora/temp-lora/blob/HEAD/trainer/acc_pg19_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70a4a56320ca2468"}},{"code_sha256_prefix":"15ec368adac197a9","entry":"generate_once","repo":"temporarylora/temp-lora","repo_kind":"official","path":"trainer/acc_complete_example_trainer.py","file_url":"https://github.com/temporarylora/temp-lora/blob/HEAD/trainer/acc_complete_example_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15ec368adac197a9"}},{"code_sha256_prefix":"0dc9f3b4f24ba5f4","entry":"train_once","repo":"temporarylora/temp-lora","repo_kind":"official","path":"trainer/acc_pg19_trainer.py","file_url":"https://github.com/temporarylora/temp-lora/blob/HEAD/trainer/acc_pg19_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0dc9f3b4f24ba5f4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}