{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evolving-and-understanding-sparse-deep-neural","title":"A Brain-inspired Algorithm for Training Highly Sparse Neural Networks","arxiv_id":"1903.07138","date":"2019-03-17","proceeding":null,"authors":["Zahra Atashgahi","Joost Pieterse","Shiwei Liu","Decebal Constantin Mocanu","Raymond Veldhuis","Mykola Pechenizkiy"],"abstract":"Sparse neural networks attract increasing interest as they exhibit comparable performance to their dense counterparts while being computationally efficient. Pruning the dense neural networks is among the most widely used methods to obtain a sparse neural network. Driven by the high training cost of such methods that can be unaffordable for a low-resource device, training sparse neural networks sparsely from scratch has recently gained attention. However, existing sparse training algorithms suffer from various issues, including poor performance in high sparsity scenarios, computing dense gradient information during training, or pure random topology search. In this paper, inspired by the evolution of the biological brain and the Hebbian learning theory, we present a new sparse training approach that evolves sparse neural networks according to the behavior of neurons in the network. Concretely, by exploiting the cosine similarity metric to measure the importance of the connections, our proposed method, Cosine similarity-based and Random Topology Exploration (CTRE), evolves the topology of sparse neural networks by adding the most important connections to the network without calculating dense gradient in the backward. We carried out different experiments on eight datasets, including tabular, image, and text datasets, and demonstrate that our proposed method outperforms several state-of-the-art sparse training algorithms in extremely sparse neural networks by a large gap. The implementation code is available on https://github.com/zahraatashgahi/CTRE","url_abs":"https://arxiv.org/abs/1903.07138v3","url_pdf":"https://arxiv.org/pdf/1903.07138v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evolving-and-understanding-sparse-deep-neural","repo_url":"https://github.com/joostPieterse/CosineSET","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"evolving-and-understanding-sparse-deep-neural","repo_url":"https://github.com/zahraatashgahi/ctre","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"learning-theory","task_name":"Learning Theory"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1903.07138","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1903.07138"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joostPieterse/CosineSET","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zahraatashgahi/ctre","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran_honours":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"1cb2d51570c25ff3","entry":"createWeightsMask","repo":"joostPieterse/CosineSET","repo_kind":"official","path":"cosine_set_fixprob_dense_mlp.py","file_url":"https://github.com/joostPieterse/CosineSET/blob/HEAD/cosine_set_fixprob_dense_mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1cb2d51570c25ff3"}},{"code_sha256_prefix":"09540a28779b66ce","entry":"find_first_pos","repo":"joostPieterse/CosineSET","repo_kind":"official","path":"cosine_set_fixprob_dense_mlp.py","file_url":"https://github.com/joostPieterse/CosineSET/blob/HEAD/cosine_set_fixprob_dense_mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09540a28779b66ce"}},{"code_sha256_prefix":"6c42c3044030b535","entry":"find_last_pos","repo":"joostPieterse/CosineSET","repo_kind":"official","path":"cosine_set_fixprob_dense_mlp.py","file_url":"https://github.com/joostPieterse/CosineSET/blob/HEAD/cosine_set_fixprob_dense_mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c42c3044030b535"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}