{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/exploring-the-benefit-of-activation-sparsity","title":"Exploring the Benefit of Activation Sparsity in Pre-training","arxiv_id":"2410.03440","date":"2024-10-04","proceeding":null,"authors":["Zhengyan Zhang","Chaojun Xiao","Qiujieli Qin","Yankai Lin","Zhiyuan Zeng","Xu Han","Zhiyuan Liu","Ruobing Xie","Maosong Sun","Jie zhou"],"abstract":"Pre-trained Transformers inherently possess the characteristic of sparse activation, where only a small fraction of the neurons are activated for each token. While sparse activation has been explored through post-training methods, its potential in pre-training remains untapped. In this work, we first study how activation properties change during pre-training. Our examination reveals that Transformers exhibit sparse activation throughout the majority of the pre-training process while the activation correlation keeps evolving as training progresses. Leveraging this observation, we propose Switchable Sparse-Dense Learning (SSD). SSD adaptively switches between the Mixtures-of-Experts (MoE) based sparse training and the conventional dense training during the pre-training process, leveraging the efficiency of sparse training and avoiding the static activation correlation of sparse training. Compared to dense training, SSD achieves comparable performance with identical model size and reduces pre-training costs. Moreover, the models trained with SSD can be directly used as MoE models for sparse inference and achieve the same performance as dense models with up to $2\\times$ faster inference speed. Codes are available at https://github.com/thunlp/moefication.","url_abs":"https://arxiv.org/abs/2410.03440v1","url_pdf":"https://arxiv.org/pdf/2410.03440v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"exploring-the-benefit-of-activation-sparsity","repo_url":"https://github.com/thunlp/moefication","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"moe","method_name":"MoE"},{"method_slug":"non-maximum-suppression","method_name":"Non Maximum Suppression"},{"method_slug":"ssd","method_name":"SSD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.03440","atlas_url":"https://app.syntology.ai/?focus=2410.03440","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.03440"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/thunlp/moefication","reach":{"status":"ok"}}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"703e3ecf17cfa606","entry":"cal_intertia_","repo":"thunlp/moefication","repo_kind":"official","path":"faster_moefication/moefication.py","file_url":"https://github.com/thunlp/moefication/blob/HEAD/faster_moefication/moefication.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"703e3ecf17cfa606"}},{"code_sha256_prefix":"0a1340701fa70104","entry":"get_layer_num","repo":"thunlp/moefication","repo_kind":"official","path":"moefication/utils.py","file_url":"https://github.com/thunlp/moefication/blob/HEAD/moefication/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a1340701fa70104"}},{"code_sha256_prefix":"158405f0f4beeff8","entry":"load_ffn_weight","repo":"thunlp/moefication","repo_kind":"official","path":"moefication/utils.py","file_url":"https://github.com/thunlp/moefication/blob/HEAD/moefication/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"158405f0f4beeff8"}},{"code_sha256_prefix":"ceb1bef670471d4e","entry":"load_hidden_states","repo":"thunlp/moefication","repo_kind":"official","path":"moefication/utils.py","file_url":"https://github.com/thunlp/moefication/blob/HEAD/moefication/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ceb1bef670471d4e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}