{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-00454","title":"Universal Properties of Activation Sparsity in Modern Large Language Models","arxiv_id":"2509.00454","date":"2025-08-30","proceeding":"ICLR","authors":["Filip Szatkowski","Patryk Będkowski","Alessio Devoto","Jan Dubiński","Pasquale Minervini","Mikołaj Piórczyński","Simone Scardapane","Bartosz Wójcik"],"abstract":"Activation sparsity is an intriguing property of deep neural networks that has been extensively studied in ReLU-based models, due to its advantages for efficiency, robustness, and interpretability. However, methods relying on exact zero activations do not directly apply to modern Large Language Models (LLMs), leading to fragmented, model-specific strategies for LLM activation sparsity and a gap in its general understanding. In this work, we introduce a general framework for evaluating sparsity robustness in contemporary LLMs and conduct a systematic investigation of this phenomenon in their feedforward~(FFN) layers. Our results uncover universal properties of activation sparsity across diverse model families and scales. Importantly, we observe that the potential for effective activation sparsity grows with model size, highlighting its increasing relevance as models scale. Furthermore, we present the first study of activation sparsity in diffusion-based LLMs. Overall, our work provides a comprehensive perspective and practical guidance for harnessing activation sparsity in LLM design and acceleration.","url_abs":"https://arxiv.org/abs/2509.00454","url_pdf":"https://arxiv.org/pdf/2509.00454","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2509.00454","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.00454"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/fszatkowski/activation-sparsity-benchmarking","reach":null}],"summary":{"ran":4,"ran_violates":1,"ran_draft_wrong":4,"ran_fixture":2,"ran_honours":1,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":16,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"77a3db6e1268aa54","entry":"RunningEffectiveRankStats","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77a3db6e1268aa54"}},{"code_sha256_prefix":"a7b9124211fff3db","entry":"RunningSparsityStats","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a7b9124211fff3db"}},{"code_sha256_prefix":"d9b039820ab38a17","entry":"SampleSparsityStats","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9b039820ab38a17"}},{"code_sha256_prefix":"578f4efbd874cef5","entry":"SequenceSparsityStats","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"578f4efbd874cef5"}},{"code_sha256_prefix":"b14dd9fa7608860c","entry":"get_maxp_mask","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b14dd9fa7608860c"}},{"code_sha256_prefix":"0de0b1467eee9d08","entry":"get_topk_mask","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0de0b1467eee9d08"}},{"code_sha256_prefix":"1a2667dea331d9a1","entry":"get_topp_mask","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a2667dea331d9a1"}},{"code_sha256_prefix":"ce0fa297534f46b3","entry":"parse_hooks_effective_rank_stats","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce0fa297534f46b3"}},{"code_sha256_prefix":"c28caf5fb9e3c1a5","entry":"parse_hooks_sparsity_stats","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c28caf5fb9e3c1a5"}},{"code_sha256_prefix":"f80740a72f5cbe92","entry":"parse_layers_to_sparsify","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f80740a72f5cbe92"}},{"code_sha256_prefix":"5fabcf44742faa65","entry":"read_sparsification_config","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5fabcf44742faa65"}},{"code_sha256_prefix":"91822641d73a0bdb","entry":"sparsify_tensor","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91822641d73a0bdb"}},{"code_sha256_prefix":"46b8f35f4b47c670","entry":"InputSparsificationHook","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46b8f35f4b47c670"}},{"code_sha256_prefix":"0419fcca5c34f730","entry":"OutputSparsificationHook","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0419fcca5c34f730"}},{"code_sha256_prefix":"1f447e960004837a","entry":"SparsificationHook","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f447e960004837a"}},{"code_sha256_prefix":"f5ac82d029faba0a","entry":"SparsificationManager","repo":"fszatkowski/activation-sparsity-benchmarking","repo_kind":"found_in_text","path":"sparsified_harness/lm_eval/sparsification_manager.py","file_url":"https://github.com/fszatkowski/activation-sparsity-benchmarking/blob/HEAD/sparsified_harness/lm_eval/sparsification_manager.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f5ac82d029faba0a"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}