{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/minference-1-0-accelerating-pre-filling-for","title":"MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention","arxiv_id":"2407.02490","date":"2024-07-02","proceeding":null,"authors":["Huiqiang Jiang","Yucheng Li","Chengruidong Zhang","Qianhui Wu","Xufang Luo","Surin Ahn","Zhenhua Han","Amir H. Abdi","Dongsheng Li","Chin-Yew Lin","Yuqing Yang","Lili Qiu"],"abstract":"The computational challenges of Large Language Model (LLM) inference remain a significant barrier to their widespread deployment, especially as prompt lengths continue to increase. Due to the quadratic complexity of the attention computation, it takes 30 minutes for an 8B LLM to process a prompt of 1M tokens (i.e., the pre-filling stage) on a single A100 GPU. Existing methods for speeding up prefilling often fail to maintain acceptable accuracy or efficiency when applied to long-context LLMs. To address this gap, we introduce MInference (Milliontokens Inference), a sparse calculation method designed to accelerate pre-filling of long-sequence processing. Specifically, we identify three unique patterns in long-context attention matrices-the A-shape, Vertical-Slash, and Block-Sparsethat can be leveraged for efficient sparse computation on GPUs. We determine the optimal pattern for each attention head offline and dynamically build sparse indices based on the assigned pattern during inference. With the pattern and sparse indices, we perform efficient sparse attention calculations via our optimized GPU kernels to significantly reduce the latency in the pre-filling stage of long-context LLMs. Our proposed technique can be directly applied to existing LLMs without any modifications to the pre-training setup or additional fine-tuning. By evaluating on a wide range of downstream tasks, including InfiniteBench, RULER, PG-19, and Needle In A Haystack, and models including LLaMA-3-1M, GLM4-1M, Yi-200K, Phi-3-128K, and Qwen2-128K, we demonstrate that MInference effectively reduces inference latency by up to 10x for pre-filling on an A100, while maintaining accuracy. Our code is available at https://aka.ms/MInference.","url_abs":"https://arxiv.org/abs/2407.02490v2","url_pdf":"https://arxiv.org/pdf/2407.02490v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"minference-1-0-accelerating-pre-filling-for","repo_url":"https://github.com/microsoft/MInference","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"minference-1-0-accelerating-pre-filling-for","repo_url":"https://github.com/microsoft/LLMLingua","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.02490","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.02490"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/LLMLingua","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/MInference","reach":null}],"summary":{"ran":8,"ran_honours":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"listed":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c03b7f062823101c","entry":"filter_by_metric","repo":"microsoft/LLMLingua","repo_kind":"listed","path":"experiments/securitylingua/filter.py","file_url":"https://github.com/microsoft/LLMLingua/blob/HEAD/experiments/securitylingua/filter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c03b7f062823101c"}},{"code_sha256_prefix":"c60f40b9d6ed164e","entry":"get_pure_token","repo":"microsoft/LLMLingua","repo_kind":"listed","path":"llmlingua/utils.py","file_url":"https://github.com/microsoft/LLMLingua/blob/HEAD/llmlingua/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c60f40b9d6ed164e"}},{"code_sha256_prefix":"82091eb9827b6af7","entry":"is_begin_of_new_word","repo":"microsoft/LLMLingua","repo_kind":"listed","path":"llmlingua/utils.py","file_url":"https://github.com/microsoft/LLMLingua/blob/HEAD/llmlingua/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82091eb9827b6af7"}},{"code_sha256_prefix":"7da9282cb2dcf1ab","entry":"is_equal","repo":"microsoft/LLMLingua","repo_kind":"listed","path":"experiments/llmlingua2/data_collection/label_word.py","file_url":"https://github.com/microsoft/LLMLingua/blob/HEAD/experiments/llmlingua2/data_collection/label_word.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7da9282cb2dcf1ab"}},{"code_sha256_prefix":"5c2c5ab6b6d57f48","entry":"is_equal","repo":"microsoft/LLMLingua","repo_kind":"listed","path":"experiments/securitylingua/label_word.py","file_url":"https://github.com/microsoft/LLMLingua/blob/HEAD/experiments/securitylingua/label_word.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c2c5ab6b6d57f48"}},{"code_sha256_prefix":"10fb3b88a9539710","entry":"load_and_split_data","repo":"microsoft/LLMLingua","repo_kind":"listed","path":"experiments/securitylingua/train_roberta.py","file_url":"https://github.com/microsoft/LLMLingua/blob/HEAD/experiments/securitylingua/train_roberta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10fb3b88a9539710"}},{"code_sha256_prefix":"c22578e87f0bdd2c","entry":"replace_added_token","repo":"microsoft/LLMLingua","repo_kind":"listed","path":"llmlingua/utils.py","file_url":"https://github.com/microsoft/LLMLingua/blob/HEAD/llmlingua/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c22578e87f0bdd2c"}},{"code_sha256_prefix":"4e369dc3b4632635","entry":"setup_logging","repo":"microsoft/LLMLingua","repo_kind":"listed","path":"experiments/securitylingua/label_word.py","file_url":"https://github.com/microsoft/LLMLingua/blob/HEAD/experiments/securitylingua/label_word.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e369dc3b4632635"}},{"code_sha256_prefix":"c1c592b93449e76c","entry":"truncate_by_tokens","repo":"microsoft/MInference","repo_kind":"official","path":"experiments/infinite_bench/run_infinitebench.py","file_url":"https://github.com/microsoft/MInference/blob/HEAD/experiments/infinite_bench/run_infinitebench.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1c592b93449e76c"}},{"code_sha256_prefix":"db3017985563c2a7","entry":"truncate_input","repo":"microsoft/MInference","repo_kind":"official","path":"experiments/infinite_bench/run_infinitebench.py","file_url":"https://github.com/microsoft/MInference/blob/HEAD/experiments/infinite_bench/run_infinitebench.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db3017985563c2a7"}},{"code_sha256_prefix":"bdd16b22daad11a5","entry":"setup_wandb","repo":"microsoft/LLMLingua","repo_kind":"listed","path":"experiments/securitylingua/train_roberta.py","file_url":"https://github.com/microsoft/LLMLingua/blob/HEAD/experiments/securitylingua/train_roberta.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdd16b22daad11a5"}},{"code_sha256_prefix":"469e5cfe9fc2ef17","entry":"split_string","repo":"microsoft/LLMLingua","repo_kind":"listed","path":"experiments/securitylingua/label_word.py","file_url":"https://github.com/microsoft/LLMLingua/blob/HEAD/experiments/securitylingua/label_word.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"469e5cfe9fc2ef17"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}