{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/giraffe-adventures-in-expanding-context","title":"Giraffe: Adventures in Expanding Context Lengths in LLMs","arxiv_id":"2308.10882","date":"2023-08-21","proceeding":null,"authors":["Arka Pal","Deep Karkhanis","Manley Roberts","Samuel Dooley","Arvind Sundararajan","Siddartha Naidu"],"abstract":"Modern large language models (LLMs) that rely on attention mechanisms are typically trained with fixed context lengths which enforce upper limits on the length of input sequences that they can handle at evaluation time. To use these models on sequences longer than the train-time context length, one might employ techniques from the growing family of context length extrapolation methods -- most of which focus on modifying the system of positional encodings used in the attention mechanism to indicate where tokens or activations are located in the input sequence. We conduct a wide survey of existing methods of context length extrapolation on a base LLaMA or LLaMA 2 model, and introduce some of our own design as well -- in particular, a new truncation strategy for modifying the basis for the position encoding. We test these methods using three new evaluation tasks (FreeFormQA, AlteredNumericQA, and LongChat-Lines) as well as perplexity, which we find to be less fine-grained as a measure of long context performance of LLMs. We release the three tasks publicly as datasets on HuggingFace. We discover that linear scaling is the best method for extending context length, and show that further gains can be achieved by using longer scales at evaluation time. We also discover promising extrapolation capabilities in the truncated basis. To support further research in this area, we release three new 13B parameter long-context models which we call Giraffe: 4k and 16k context models trained from base LLaMA-13B, and a 32k context model trained from base LLaMA2-13B. We also release the code to replicate our results.","url_abs":"https://arxiv.org/abs/2308.10882v1","url_pdf":"https://arxiv.org/pdf/2308.10882v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"giraffe-adventures-in-expanding-context","repo_url":"https://github.com/abacusai/long-context","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"16k","task_name":"16k"},{"task_slug":"4k","task_name":"4k"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.10882","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.10882"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abacusai/long-context","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran":7,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"abacusai/long-context","repo_kind":"official","path":"python/models/modeling_giraffe.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/models/modeling_giraffe.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"37fc3129c3d06e0c","entry":"apply_rotary_pos_emb","repo":"abacusai/long-context","repo_kind":"official","path":"python/models/modeling_giraffe.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/models/modeling_giraffe.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37fc3129c3d06e0c"}},{"code_sha256_prefix":"0e92b47ad30b6534","entry":"chunk","repo":"abacusai/long-context","repo_kind":"official","path":"python/train/finetune_context.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/train/finetune_context.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e92b47ad30b6534"}},{"code_sha256_prefix":"03e99c761c545617","entry":"duplicate_interleave","repo":"abacusai/long-context","repo_kind":"official","path":"python/models/xpos.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/models/xpos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"03e99c761c545617"}},{"code_sha256_prefix":"ff55f4ed6ff9941f","entry":"fixed_pos_embedding","repo":"abacusai/long-context","repo_kind":"official","path":"python/models/xpos.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/models/xpos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff55f4ed6ff9941f"}},{"code_sha256_prefix":"a0583ce41b433723","entry":"join_pages","repo":"abacusai/long-context","repo_kind":"official","path":"python/train/finetune_context.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/train/finetune_context.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a0583ce41b433723"}},{"code_sha256_prefix":"769944cde562a1ab","entry":"load_testcases","repo":"abacusai/long-context","repo_kind":"official","path":"python/eval/longeval/utils.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/eval/longeval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"769944cde562a1ab"}},{"code_sha256_prefix":"368e8872aa996b9e","entry":"rotate_every_two","repo":"abacusai/long-context","repo_kind":"official","path":"python/models/xpos.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/models/xpos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"368e8872aa996b9e"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"abacusai/long-context","repo_kind":"official","path":"python/models/modeling_giraffe.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/models/modeling_giraffe.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"70016ef148e9a350","entry":"truncate_frequency","repo":"abacusai/long-context","repo_kind":"official","path":"python/models/interpolate.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/models/interpolate.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"70016ef148e9a350"}},{"code_sha256_prefix":"cebd84ad2b9da0a0","entry":"batch_apply_rotary_pos_emb","repo":"abacusai/long-context","repo_kind":"official","path":"python/models/interpolate.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/models/interpolate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cebd84ad2b9da0a0"}},{"code_sha256_prefix":"f1195d8a23d10b75","entry":"get_output_dir","repo":"abacusai/long-context","repo_kind":"official","path":"python/eval/longeval/utils.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/eval/longeval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f1195d8a23d10b75"}},{"code_sha256_prefix":"69b3ed047105ab30","entry":"tokenize_and_chunk","repo":"abacusai/long-context","repo_kind":"official","path":"python/train/finetune_context.py","file_url":"https://github.com/abacusai/long-context/blob/HEAD/python/train/finetune_context.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"69b3ed047105ab30"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}