{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/focused-transformer-contrastive-training-for","title":"Focused Transformer: Contrastive Training for Context Scaling","arxiv_id":"2307.03170","date":"2023-07-06","proceeding":"NeurIPS 2023 11","authors":["Szymon Tworkowski","Konrad Staniszewski","Mikołaj Pacek","Yuhuai Wu","Henryk Michalewski","Piotr Miłoś"],"abstract":"Large language models have an exceptional capability to incorporate new information in a contextual manner. However, the full potential of such an approach is often restrained due to a limitation in the effective context length. One solution to this issue is to endow an attention layer with access to an external memory, which comprises of (key, value) pairs. Yet, as the number of documents increases, the proportion of relevant keys to irrelevant ones decreases, leading the model to focus more on the irrelevant keys. We identify a significant challenge, dubbed the distraction issue, where keys linked to different semantic values might overlap, making them hard to distinguish. To tackle this problem, we introduce the Focused Transformer (FoT), a technique that employs a training process inspired by contrastive learning. This novel approach enhances the structure of the (key, value) space, enabling an extension of the context length. Our method allows for fine-tuning pre-existing, large-scale models to lengthen their effective context. This is demonstrated by our fine-tuning of $3B$ and $7B$ OpenLLaMA checkpoints. The resulting models, which we name LongLLaMA, exhibit advancements in tasks requiring a long context. We further illustrate that our LongLLaMA models adeptly manage a $256 k$ context length for passkey retrieval.","url_abs":"https://arxiv.org/abs/2307.03170v2","url_pdf":"https://arxiv.org/pdf/2307.03170v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"focused-transformer-contrastive-training-for","repo_url":"https://github.com/cstankonrad/long_llama","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.03170","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.03170"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/CStanKonrad/long_llama","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cstankonrad/long_llama","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"ran_draft_wrong":3,"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":14,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cc2ba1b1b0c83f6c","entry":"match_keywords","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"fot_continued_pretraining/EasyLM/models/llama/convert_easylm_to_hf.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/fot_continued_pretraining/EasyLM/models/llama/convert_easylm_to_hf.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc2ba1b1b0c83f6c"}},{"code_sha256_prefix":"43de84e441595b25","entry":"mem_apply_update","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"src/longllama_utils.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/src/longllama_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"43de84e441595b25"}},{"code_sha256_prefix":"3544c0f9d7c6110f","entry":"metrics_assign_group","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"instruction_fine_tuning/utils.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/instruction_fine_tuning/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3544c0f9d7c6110f"}},{"code_sha256_prefix":"5a004e0660382250","entry":"non_numeric_to_str","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"instruction_fine_tuning/utils.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/instruction_fine_tuning/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a004e0660382250"}},{"code_sha256_prefix":"1d636c739a91d422","entry":"precompute_freqs_cis","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"fot_continued_pretraining/EasyLM/models/llama/llama_model.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/fot_continued_pretraining/EasyLM/models/llama/llama_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1d636c739a91d422"}},{"code_sha256_prefix":"c5bcf01d18bba63d","entry":"read_json","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"fot_continued_pretraining/EasyLM/models/llama/convert_easylm_to_hf.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/fot_continued_pretraining/EasyLM/models/llama/convert_easylm_to_hf.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c5bcf01d18bba63d"}},{"code_sha256_prefix":"a13c903093d13c1f","entry":"rotate_as_if_first","repo":"cstankonrad/long_llama","repo_kind":"official","path":"src/modeling_longllama.py","file_url":"https://github.com/cstankonrad/long_llama/blob/HEAD/src/modeling_longllama.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a13c903093d13c1f"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"src/modeling_longllama.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/src/modeling_longllama.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"feba6513abb9fb9c","entry":"rotate_one","repo":"cstankonrad/long_llama","repo_kind":"official","path":"src/modeling_longllama.py","file_url":"https://github.com/cstankonrad/long_llama/blob/HEAD/src/modeling_longllama.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"feba6513abb9fb9c"}},{"code_sha256_prefix":"a305ca669a9cacf3","entry":"apply_rotary_emb","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"fot_continued_pretraining/EasyLM/models/llama/llama_model.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/fot_continued_pretraining/EasyLM/models/llama/llama_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a305ca669a9cacf3"}},{"code_sha256_prefix":"fd8c57f88ddcfa62","entry":"apply_rotary_emb_as_fst","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"fot_continued_pretraining/EasyLM/models/llama/llama_model.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/fot_continued_pretraining/EasyLM/models/llama/llama_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd8c57f88ddcfa62"}},{"code_sha256_prefix":"21a812da7fd6cdd2","entry":"get_padding_token","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"instruction_fine_tuning/data_processing.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/instruction_fine_tuning/data_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"21a812da7fd6cdd2"}},{"code_sha256_prefix":"c2234099a81ca2e6","entry":"inverse_permute","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"fot_continued_pretraining/EasyLM/models/llama/convert_hf_to_easylm.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/fot_continued_pretraining/EasyLM/models/llama/convert_hf_to_easylm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c2234099a81ca2e6"}},{"code_sha256_prefix":"630a45cda4fbc26b","entry":"tokenize_text_no_special_tokens","repo":"CStanKonrad/long_llama","repo_kind":"official","path":"instruction_fine_tuning/data_processing.py","file_url":"https://github.com/CStanKonrad/long_llama/blob/HEAD/instruction_fine_tuning/data_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"630a45cda4fbc26b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}