{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/topological-insights-in-sparse-neural","title":"Topological Insights into Sparse Neural Networks","arxiv_id":"2006.14085","date":"2020-06-24","proceeding":null,"authors":["Shiwei Liu","Tim Van der Lee","Anil Yaman","Zahra Atashgahi","Davide Ferraro","Ghada Sokar","Mykola Pechenizkiy","Decebal Constantin Mocanu"],"abstract":"Sparse neural networks are effective approaches to reduce the resource requirements for the deployment of deep neural networks. Recently, the concept of adaptive sparse connectivity, has emerged to allow training sparse neural networks from scratch by optimizing the sparse structure during training. However, comparing different sparse topologies and determining how sparse topologies evolve during training, especially for the situation in which the sparse structure optimization is involved, remain as challenging open questions. This comparison becomes increasingly complex as the number of possible topological comparisons increases exponentially with the size of networks. In this work, we introduce an approach to understand and compare sparse neural network topologies from the perspective of graph theory. We first propose Neural Network Sparse Topology Distance (NNSTD) to measure the distance between different sparse neural networks. Further, we demonstrate that sparse neural networks can outperform over-parameterized models in terms of performance, even without any further structure optimization. To the end, we also show that adaptive sparse connectivity can always unveil a plenitude of sparse sub-networks with very different topologies which outperform the dense model, by quantifying and comparing their topological evolutionary processes. The latter findings complement the Lottery Ticket Hypothesis by showing that there is a much more efficient and robust way to find \"winning tickets\". Altogether, our results start enabling a better theoretical understanding of sparse neural networks, and demonstrate the utility of using graph theory to analyze them.","url_abs":"https://arxiv.org/abs/2006.14085v2","url_pdf":"https://arxiv.org/pdf/2006.14085v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"topological-insights-in-sparse-neural","repo_url":"https://github.com/adubowski/set-mlp-keras","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"topological-insights-in-sparse-neural","repo_url":"https://github.com/dcmocanu/sparse-evolutionary-artificial-neural-networks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2006.14085","atlas_url":"https://app.syntology.ai/?focus=2006.14085","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.14085"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dcmocanu/sparse-evolutionary-artificial-neural-networks","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adubowski/set-mlp-keras","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_honours":2,"ran":1,"unverified":2},"by_repo_kind":{"listed":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"d1dd6b1240c82834","entry":"createSparseWeights","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d1dd6b1240c82834"}},{"code_sha256_prefix":"1cb2d51570c25ff3","entry":"createWeightsMask","repo":"adubowski/set-mlp-keras","repo_kind":"listed","path":"SET-MLP-Keras-Weights-Mask/fixprob_mlp_keras_cifar10.py","file_url":"https://github.com/adubowski/set-mlp-keras/blob/HEAD/SET-MLP-Keras-Weights-Mask/fixprob_mlp_keras_cifar10.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1cb2d51570c25ff3"}},{"code_sha256_prefix":"09540a28779b66ce","entry":"find_first_pos","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"09540a28779b66ce"}},{"code_sha256_prefix":"6c42c3044030b535","entry":"find_last_pos","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"6c42c3044030b535"}},{"code_sha256_prefix":"e84c822cd2101b70","entry":"sreluPlot","repo":"adubowski/set-mlp-keras","repo_kind":"listed","path":"SET-MLP-Keras-Weights-Mask/plot_functions.py","file_url":"https://github.com/adubowski/set-mlp-keras/blob/HEAD/SET-MLP-Keras-Weights-Mask/plot_functions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e84c822cd2101b70"}},{"code_sha256_prefix":"895ca4b776fb6d6d","entry":"rolling_window","repo":"adubowski/set-mlp-keras","repo_kind":"listed","path":"SET-MLP-Keras-Weights-Mask/plot_overfitting_per_sparsity.py","file_url":"https://github.com/adubowski/set-mlp-keras/blob/HEAD/SET-MLP-Keras-Weights-Mask/plot_overfitting_per_sparsity.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"895ca4b776fb6d6d"}},{"code_sha256_prefix":"6f8448154976771e","entry":"swish","repo":"adubowski/set-mlp-keras","repo_kind":"listed","path":"SET-MLP-Keras-Weights-Mask/dense_mlp_keras_cifar10.py","file_url":"https://github.com/adubowski/set-mlp-keras/blob/HEAD/SET-MLP-Keras-Weights-Mask/dense_mlp_keras_cifar10.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f8448154976771e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}