{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/navigating-extremes-dynamic-sparsity-in-large","title":"Navigating Extremes: Dynamic Sparsity in Large Output Space","arxiv_id":"2411.03171","date":"2024-11-05","proceeding":null,"authors":["Nasib Ullah","Erik Schultheis","Mike Lasby","Yani Ioannou","Rohit Babbar"],"abstract":"In recent years, Dynamic Sparse Training (DST) has emerged as an alternative to post-training pruning for generating efficient models. In principle, DST allows for a more memory efficient training process, as it maintains sparsity throughout the entire training run. However, current DST implementations fail to capitalize on this in practice. Because sparse matrix multiplication is much less efficient than dense matrix multiplication on GPUs, most implementations simulate sparsity by masking weights. In this paper, we leverage recent advances in semi-structured sparse training to apply DST in the domain of classification with large output spaces, where memory-efficiency is paramount. With a label space of possibly millions of candidates, the classification layer alone will consume several gigabytes of memory. Switching from a dense to a fixed fan-in sparse layer updated with sparse evolutionary training (SET); however, severely hampers training convergence, especially at the largest label spaces. We find that poor gradient flow from the sparse classifier to the dense text encoder make it difficult to learn good input representations. By employing an intermediate layer or adding an auxiliary training objective, we recover most of the generalisation performance of the dense model. Overall, we demonstrate the applicability and practical benefits of DST in a challenging domain -- characterized by a highly skewed label distribution that differs substantially from typical DST benchmark datasets -- which enables end-to-end training with millions of labels on commodity hardware.","url_abs":"https://arxiv.org/abs/2411.03171v2","url_pdf":"https://arxiv.org/pdf/2411.03171v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"navigating-extremes-dynamic-sparsity-in-large","repo_url":"https://github.com/xmc-aalto/NeurIPS24-dst","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"dst","method_name":"DST"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.03171","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.03171"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xmc-aalto/NeurIPS24-dst","reach":{"status":"ok"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"90b6172fa5dbab0b","entry":"clean_str","repo":"xmc-aalto/NeurIPS24-dst","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/xmc-aalto/NeurIPS24-dst/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90b6172fa5dbab0b"}},{"code_sha256_prefix":"5f5978e283404133","entry":"collate","repo":"xmc-aalto/NeurIPS24-dst","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/xmc-aalto/NeurIPS24-dst/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f5978e283404133"}},{"code_sha256_prefix":"63eb98e4f891edda","entry":"print_trainable_parameters","repo":"xmc-aalto/NeurIPS24-dst","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/xmc-aalto/NeurIPS24-dst/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"63eb98e4f891edda"}},{"code_sha256_prefix":"edd92d3fba08fcc2","entry":"sparsification","repo":"xmc-aalto/NeurIPS24-dst","repo_kind":"official","path":"src/evaluate.py","file_url":"https://github.com/xmc-aalto/NeurIPS24-dst/blob/HEAD/src/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"edd92d3fba08fcc2"}},{"code_sha256_prefix":"7bc27d908852d57b","entry":"timeit","repo":"xmc-aalto/NeurIPS24-dst","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/xmc-aalto/NeurIPS24-dst/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7bc27d908852d57b"}},{"code_sha256_prefix":"134597c1f018c203","entry":"to_wandb_dict","repo":"xmc-aalto/NeurIPS24-dst","repo_kind":"official","path":"src/log.py","file_url":"https://github.com/xmc-aalto/NeurIPS24-dst/blob/HEAD/src/log.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"134597c1f018c203"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}