{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/discovering-the-gems-in-early-layers","title":"Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction","arxiv_id":"2409.17422","date":"2024-09-25","proceeding":null,"authors":["Zhenmei Shi","Yifei Ming","Xuan-Phi Nguyen","YIngyu Liang","Shafiq Joty"],"abstract":"Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased computational resources and latency. Our research introduces a novel approach for the long context bottleneck to accelerate LLM inference and reduce GPU memory consumption. Our research demonstrates that LLMs can identify relevant tokens in the early layers before generating answers to a query. Leveraging this insight, we propose an algorithm that uses early layers of an LLM as filters to select and compress input tokens, significantly reducing the context length for subsequent processing. Our method, GemFilter, demonstrates substantial improvements in both speed and memory efficiency compared to existing techniques, such as standard attention and SnapKV/H2O. Notably, it achieves a 2.4$\\times$ speedup and 30\\% reduction in GPU memory usage compared to SOTA methods. Evaluation on the Needle in a Haystack task shows that GemFilter significantly outperforms standard attention, SnapKV and demonstrates comparable performance on the LongBench challenge. GemFilter is simple, training-free, and broadly applicable across different LLMs. Crucially, it provides interpretability by allowing humans to inspect the selected input sequence. These findings not only offer practical benefits for LLM deployment, but also enhance our understanding of LLM internal mechanisms, paving the way for further optimizations in LLM design and inference. Our code is available at \\url{https://github.com/SalesforceAIResearch/GemFilter}.","url_abs":"https://arxiv.org/abs/2409.17422v1","url_pdf":"https://arxiv.org/pdf/2409.17422v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"discovering-the-gems-in-early-layers","repo_url":"https://github.com/salesforceairesearch/gemfilter","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"token-reduction","task_name":"Token Reduction"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"speed","method_name":"SPEED"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.17422","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.17422"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/salesforceairesearch/gemfilter","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":9,"ran_honours":1,"ran_draft_wrong":2,"ran_violates":1},"by_repo_kind":{"official":{"samples":13,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8339145374a403f3","entry":"build_chat","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"eval/LongBench/pred.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/eval/LongBench/pred.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8339145374a403f3"}},{"code_sha256_prefix":"b349b79d9cc2934b","entry":"count_score","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"eval/LongBench/metrics.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/eval/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b349b79d9cc2934b"}},{"code_sha256_prefix":"8023b20549593b60","entry":"get_layer_context","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"my_utils/my_generation.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/my_utils/my_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8023b20549593b60"}},{"code_sha256_prefix":"ce3c4802be48b9d2","entry":"insert_needle","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"eval/needle/utils.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/eval/needle/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce3c4802be48b9d2"}},{"code_sha256_prefix":"6af3fc7115dbdfad","entry":"load_context","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"eval/needle/utils.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/eval/needle/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6af3fc7115dbdfad"}},{"code_sha256_prefix":"74f9afa035c0616b","entry":"my_forward","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"my_utils/load_model.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/my_utils/load_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"74f9afa035c0616b"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"eval/LongBench/metrics.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/eval/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"eval/LongBench/metrics.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/eval/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"7e76d6613e17885d","entry":"parse_args","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"eval/LongBench/pred.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/eval/LongBench/pred.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e76d6613e17885d"}},{"code_sha256_prefix":"4489113b536ca6eb","entry":"post_process","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"eval/LongBench/pred.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/eval/LongBench/pred.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4489113b536ca6eb"}},{"code_sha256_prefix":"7492f636f610e78b","entry":"recover_layer","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"my_utils/my_generation.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/my_utils/my_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7492f636f610e78b"}},{"code_sha256_prefix":"9fbfc530cceb902f","entry":"reduce_layer","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"my_utils/my_generation.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/my_utils/my_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9fbfc530cceb902f"}},{"code_sha256_prefix":"1d164ab515de6ff5","entry":"standard_dis_index","repo":"salesforceairesearch/gemfilter","repo_kind":"official","path":"my_baseline/GemFilter/gem_filter_utils.py","file_url":"https://github.com/salesforceairesearch/gemfilter/blob/HEAD/my_baseline/GemFilter/gem_filter_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1d164ab515de6ff5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}