{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-controlled-study-on-long-context-extension","title":"A Controlled Study on Long Context Extension and Generalization in LLMs","arxiv_id":"2409.12181","date":"2024-09-18","proceeding":null,"authors":["Yi Lu","Jing Nathan Yan","Songlin Yang","Justin T. Chiu","Siyu Ren","Fei Yuan","Wenting Zhao","Zhiyong Wu","Alexander M. Rush"],"abstract":"Broad textual understanding and in-context learning require language models that utilize full document contexts. Due to the implementation challenges associated with directly training long-context models, many methods have been proposed for extending models to handle long contexts. However, owing to differences in data and model classes, it has been challenging to compare these approaches, leading to uncertainty as to how to evaluate long-context performance and whether it differs from standard evaluation. We implement a controlled protocol for extension methods with a standardized evaluation, utilizing consistent base models and extension data. Our study yields several insights into long-context behavior. First, we reaffirm the critical role of perplexity as a general-purpose performance indicator even in longer-context tasks. Second, we find that current approximate attention methods systematically underperform across long-context tasks. Finally, we confirm that exact fine-tuning based methods are generally effective within the range of their extension, whereas extrapolation remains challenging. All codebases, models, and checkpoints will be made available open-source, promoting transparency and facilitating further research in this critical area of AI development.","url_abs":"https://arxiv.org/abs/2409.12181v2","url_pdf":"https://arxiv.org/pdf/2409.12181v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-controlled-study-on-long-context-extension","repo_url":"https://github.com/leooyii/lceg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.12181","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.12181"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leooyii/lceg","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_honours":2,"ran_draft_wrong":4,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"2fce765e480061f7","entry":"add_mem_tokens","repo":"leooyii/lceg","repo_kind":"official","path":"continuous_finetuning/fine-tune.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/continuous_finetuning/fine-tune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fce765e480061f7"}},{"code_sha256_prefix":"b349b79d9cc2934b","entry":"count_score","repo":"leooyii/lceg","repo_kind":"official","path":"longbench/metrics.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/longbench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b349b79d9cc2934b"}},{"code_sha256_prefix":"7d8751620f2e23ff","entry":"evaluate","repo":"leooyii/lceg","repo_kind":"official","path":"eval_perplexity/eval_lm_infinite.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/eval_perplexity/eval_lm_infinite.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d8751620f2e23ff"}},{"code_sha256_prefix":"51ee91de5e3fdcaa","entry":"iceildiv","repo":"leooyii/lceg","repo_kind":"official","path":"eval_perplexity/eval_clex.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/eval_perplexity/eval_clex.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"51ee91de5e3fdcaa"}},{"code_sha256_prefix":"2b06c68ea29c8f3b","entry":"load_model_and_tokenizer","repo":"leooyii/lceg","repo_kind":"official","path":"longbench/pred.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/longbench/pred.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b06c68ea29c8f3b"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"leooyii/lceg","repo_kind":"official","path":"longbench/metrics.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/longbench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"leooyii/lceg","repo_kind":"official","path":"longbench/metrics.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/longbench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"ca066a4c088c8cf5","entry":"parse_args","repo":"leooyii/lceg","repo_kind":"official","path":"longbench/eval_trec.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/longbench/eval_trec.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca066a4c088c8cf5"}},{"code_sha256_prefix":"ca70f1c4db4f1296","entry":"parse_args","repo":"leooyii/lceg","repo_kind":"official","path":"longbench/pred.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/longbench/pred.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca70f1c4db4f1296"}},{"code_sha256_prefix":"d2c659b7707d28bf","entry":"reshape_fn","repo":"leooyii/lceg","repo_kind":"official","path":"continuous_finetuning/fine-tune.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/continuous_finetuning/fine-tune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2c659b7707d28bf"}},{"code_sha256_prefix":"ad4c52b7c7083143","entry":"tokenize_fn","repo":"leooyii/lceg","repo_kind":"official","path":"continuous_finetuning/fine-tune.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/continuous_finetuning/fine-tune.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad4c52b7c7083143"}},{"code_sha256_prefix":"708bed3133d01060","entry":"evaluate","repo":"leooyii/lceg","repo_kind":"official","path":"eval_perplexity/eval_clex.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/eval_perplexity/eval_clex.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"708bed3133d01060"}},{"code_sha256_prefix":"d69813e43224dc01","entry":"evaluate","repo":"leooyii/lceg","repo_kind":"official","path":"eval_perplexity/eval_landmark.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/eval_perplexity/eval_landmark.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d69813e43224dc01"}},{"code_sha256_prefix":"ac5698be4313ef23","entry":"get_pred","repo":"leooyii/lceg","repo_kind":"official","path":"longbench/pred.py","file_url":"https://github.com/leooyii/lceg/blob/HEAD/longbench/pred.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac5698be4313ef23"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}