{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-sparse-networks-using-targeted","title":"Learning Sparse Networks Using Targeted Dropout","arxiv_id":"1905.13678","date":"2019-05-31","proceeding":null,"authors":["Aidan N. Gomez","Ivan Zhang","Siddhartha Rao Kamalakara","Divyam Madaan","Kevin Swersky","Yarin Gal","Geoffrey E. Hinton"],"abstract":"Neural networks are easier to optimise when they have many more weights than are required for modelling the mapping from inputs to outputs. This suggests a two-stage learning procedure that first learns a large net and then prunes away connections or hidden units. But standard training does not necessarily encourage nets to be amenable to pruning. We introduce targeted dropout, a method for training a neural network so that it is robust to subsequent pruning. Before computing the gradients for each weight update, targeted dropout stochastically selects a set of units or weights to be dropped using a simple self-reinforcing sparsity criterion and then computes the gradients for the remaining weights. The resulting network is robust to post hoc pruning of weights or units that frequently occur in the dropped sets. The method improves upon more complicated sparsifying regularisers while being simple to implement and easy to tune.","url_abs":"https://arxiv.org/abs/1905.13678v5","url_pdf":"https://arxiv.org/pdf/1905.13678v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-sparse-networks-using-targeted","repo_url":"https://github.com/for-ai/TD","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"learning-sparse-networks-using-targeted","repo_url":"https://github.com/andrijazz/pruning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"network-pruning","task_name":"Network Pruning"},{"task_slug":"neural-network-compression","task_name":"Neural Network Compression"}],"methods":[{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"targeted-dropout","method_name":"Targeted Dropout"}],"datasets_introduced":[],"methods_introduced":[{"slug":"targeted-dropout","name":"Targeted Dropout","full_name":"Targeted Dropout"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1905.13678","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1905.13678"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/andrijazz/pruning","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/for-ai/TD","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"f7c6faa936aaf439","entry":"matrix_topk","repo":"andrijazz/pruning","repo_kind":"listed","path":"models/pruning.py","file_url":"https://github.com/andrijazz/pruning/blob/HEAD/models/pruning.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7c6faa936aaf439"}},{"code_sha256_prefix":"e04b0413980572e2","entry":"unit_pruning","repo":"andrijazz/pruning","repo_kind":"listed","path":"models/pruning.py","file_url":"https://github.com/andrijazz/pruning/blob/HEAD/models/pruning.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e04b0413980572e2"}},{"code_sha256_prefix":"262bcd87b07100ff","entry":"weight_pruning","repo":"andrijazz/pruning","repo_kind":"listed","path":"models/pruning.py","file_url":"https://github.com/andrijazz/pruning/blob/HEAD/models/pruning.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"262bcd87b07100ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}