{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sparser-better-deeper-stronger-improving","title":"Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initialization","arxiv_id":"2406.01755","date":"2024-06-03","proceeding":null,"authors":["Aleksandra Irena Nowak","Łukasz Gniecki","Filip Szatkowski","Jacek Tabor"],"abstract":"Static sparse training aims to train sparse models from scratch, achieving remarkable results in recent years. A key design choice is given by the sparse initialization, which determines the trainable sub-network through a binary mask. Existing methods mainly select such mask based on a predefined dense initialization. Such an approach may not efficiently leverage the mask's potential impact on the optimization. An alternative direction, inspired by research into dynamical isometry, is to introduce orthogonality in the sparse subnetwork, which helps in stabilizing the gradient signal. In this work, we propose Exact Orthogonal Initialization (EOI), a novel sparse orthogonal initialization scheme based on composing random Givens rotations. Contrary to other existing approaches, our method provides exact (not approximated) orthogonality and enables the creation of layers with arbitrary densities. We demonstrate the superior effectiveness and efficiency of EOI through experiments, consistently outperforming common sparse initialization techniques. Our method enables training highly sparse 1000-layer MLP and CNN networks without residual connections or normalization techniques, emphasizing the crucial role of weight initialization in static sparse training alongside sparse mask selection. The code is available at https://github.com/woocash2/sparser-better-deeper-stronger","url_abs":"https://arxiv.org/abs/2406.01755v1","url_pdf":"https://arxiv.org/pdf/2406.01755v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sparser-better-deeper-stronger-improving","repo_url":"https://github.com/woocash2/sparser-better-deeper-stronger","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.01755","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.01755"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/woocash2/sparser-better-deeper-stronger","reach":null}],"summary":{"ran":1,"ran_draft_wrong":2,"ran_fixture":1,"ran_honours":3,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a94415e9d75ec694","entry":"MaxSparseOrtho","repo":"woocash2/sparser-better-deeper-stronger","repo_kind":"official","path":"models/initializers.py","file_url":"https://github.com/woocash2/sparser-better-deeper-stronger/blob/HEAD/models/initializers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a94415e9d75ec694"}},{"code_sha256_prefix":"6d5e7b911f34a364","entry":"apply_givens","repo":"woocash2/sparser-better-deeper-stronger","repo_kind":"official","path":"models/initializers.py","file_url":"https://github.com/woocash2/sparser-better-deeper-stronger/blob/HEAD/models/initializers.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d5e7b911f34a364"}},{"code_sha256_prefix":"862e590d44e6482f","entry":"block_convolution","repo":"woocash2/sparser-better-deeper-stronger","repo_kind":"official","path":"models/initializers.py","file_url":"https://github.com/woocash2/sparser-better-deeper-stronger/blob/HEAD/models/initializers.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"862e590d44e6482f"}},{"code_sha256_prefix":"255813605eda293f","entry":"block_orthogonal","repo":"woocash2/sparser-better-deeper-stronger","repo_kind":"official","path":"models/initializers.py","file_url":"https://github.com/woocash2/sparser-better-deeper-stronger/blob/HEAD/models/initializers.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"255813605eda293f"}},{"code_sha256_prefix":"c9d3223ca0b9b82b","entry":"get_id","repo":"woocash2/sparser-better-deeper-stronger","repo_kind":"official","path":"models/initializers.py","file_url":"https://github.com/woocash2/sparser-better-deeper-stronger/blob/HEAD/models/initializers.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9d3223ca0b9b82b"}},{"code_sha256_prefix":"72ab9f911436466e","entry":"max_sparse_ortho","repo":"woocash2/sparser-better-deeper-stronger","repo_kind":"official","path":"models/initializers.py","file_url":"https://github.com/woocash2/sparser-better-deeper-stronger/blob/HEAD/models/initializers.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72ab9f911436466e"}},{"code_sha256_prefix":"09acf4b2839c5768","entry":"max_sparse_ortho_copy","repo":"woocash2/sparser-better-deeper-stronger","repo_kind":"official","path":"models/initializers.py","file_url":"https://github.com/woocash2/sparser-better-deeper-stronger/blob/HEAD/models/initializers.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09acf4b2839c5768"}},{"code_sha256_prefix":"268bcc9ec2fa3921","entry":"Initialization","repo":"woocash2/sparser-better-deeper-stronger","repo_kind":"official","path":"models/initializers.py","file_url":"https://github.com/woocash2/sparser-better-deeper-stronger/blob/HEAD/models/initializers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"268bcc9ec2fa3921"}},{"code_sha256_prefix":"7beb966edbaf3d0a","entry":"get_col","repo":"woocash2/sparser-better-deeper-stronger","repo_kind":"official","path":"models/initializers.py","file_url":"https://github.com/woocash2/sparser-better-deeper-stronger/blob/HEAD/models/initializers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7beb966edbaf3d0a"}},{"code_sha256_prefix":"9ad8613793ceb8a2","entry":"symmetric_projection","repo":"woocash2/sparser-better-deeper-stronger","repo_kind":"official","path":"models/initializers.py","file_url":"https://github.com/woocash2/sparser-better-deeper-stronger/blob/HEAD/models/initializers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ad8613793ceb8a2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}