{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/shadowllm-predictor-based-contextual-sparsity","title":"ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models","arxiv_id":"2406.16635","date":"2024-06-24","proceeding":null,"authors":["Yash Akhauri","Ahmed F AbouElhamayed","Jordan Dotzel","Zhiru Zhang","Alexander M Rush","Safeen Huda","Mohamed S Abdelfattah"],"abstract":"The high power consumption and latency-sensitive deployments of large language models (LLMs) have motivated efficiency techniques like quantization and sparsity. Contextual sparsity, where the sparsity pattern is input-dependent, is crucial in LLMs because the permanent removal of attention heads or neurons from LLMs can significantly degrade accuracy. Prior work has attempted to model contextual sparsity using neural networks trained to predict activation magnitudes, which can be used to dynamically prune structures with low predicted activation magnitude. In this paper, we look beyond magnitude-based pruning criteria to assess attention head and neuron importance in LLMs. We develop a novel predictor called ShadowLLM, which can shadow the LLM behavior and enforce better sparsity patterns, resulting in over 15% improvement in end-to-end accuracy compared to prior methods. In addition, ShadowLLM achieves up to a 20% speed-up over the state-of-the-art DejaVu framework. These enhancements are validated on Llama-2 and OPT models with up to 30 billion parameters. Our code is available at \\href{https://github.com/abdelfattah-lab/shadow_llm/}{ShadowLLM}.","url_abs":"https://arxiv.org/abs/2406.16635v2","url_pdf":"https://arxiv.org/pdf/2406.16635v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"shadowllm-predictor-based-contextual-sparsity","repo_url":"https://github.com/abdelfattah-lab/shadow_llm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"opt","method_name":"OPT"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.16635","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.16635"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abdelfattah-lab/shadow_llm","reach":{"status":"ok"}}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"360b2f47724bdb49","entry":"combine_dicts","repo":"abdelfattah-lab/shadow_llm","repo_kind":"official","path":"llm-interpret/lm-evaluation-harness/combine_generator.py","file_url":"https://github.com/abdelfattah-lab/shadow_llm/blob/HEAD/llm-interpret/lm-evaluation-harness/combine_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"360b2f47724bdb49"}},{"code_sha256_prefix":"ea06eaae4fc1eaf0","entry":"hash_args","repo":"abdelfattah-lab/shadow_llm","repo_kind":"official","path":"llm-interpret/lm-evaluation-harness/lm_eval/base.py","file_url":"https://github.com/abdelfattah-lab/shadow_llm/blob/HEAD/llm-interpret/lm-evaluation-harness/lm_eval/base.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea06eaae4fc1eaf0"}},{"code_sha256_prefix":"ddfb29c47d101337","entry":"calculate_flops","repo":"abdelfattah-lab/shadow_llm","repo_kind":"official","path":"llm-interpret/lm-evaluation-harness/ablation_studies/ablation_predmodel_bleb1eseq/perf_model.py","file_url":"https://github.com/abdelfattah-lab/shadow_llm/blob/HEAD/llm-interpret/lm-evaluation-harness/ablation_studies/ablation_predmodel_bleb1eseq/perf_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddfb29c47d101337"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}