{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-compact-representations-of-neural","title":"Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)","arxiv_id":"2110.00684","date":"2021-10-01","proceeding":"NeurIPS 2021 12","authors":["Jie Bu","Arka Daw","M. Maruf","Anuj Karpatne"],"abstract":"A central goal in deep learning is to learn compact representations of features at every layer of a neural network, which is useful for both unsupervised representation learning and structured network pruning. While there is a growing body of work in structured pruning, current state-of-the-art methods suffer from two key limitations: (i) instability during training, and (ii) need for an additional step of fine-tuning, which is resource-intensive. At the core of these limitations is the lack of a systematic approach that jointly prunes and refines weights during training in a single stage, and does not require any fine-tuning upon convergence to achieve state-of-the-art performance. We present a novel single-stage structured pruning method termed DiscriminAtive Masking (DAM). The key intuition behind DAM is to discriminatively prefer some of the neurons to be refined during the training process, while gradually masking out other neurons. We show that our proposed DAM approach has remarkably good performance over various applications, including dimensionality reduction, recommendation system, graph representation learning, and structured pruning for image classification. We also theoretically show that the learning objective of DAM is directly related to minimizing the L0 norm of the masking layer.","url_abs":"https://arxiv.org/abs/2110.00684v1","url_pdf":"https://arxiv.org/pdf/2110.00684v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-compact-representations-of-neural","repo_url":"https://github.com/jayroxis/dam-pytorch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"dimensionality-reduction","task_name":"Dimensionality Reduction"},{"task_slug":"graph-representation-learning","task_name":"Graph Representation Learning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"network-pruning","task_name":"Network Pruning"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2110.00684","atlas_url":"https://app.syntology.ai/?focus=2110.00684","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.00684"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jayroxis/dam-pytorch","reach":null}],"summary":{"ran":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"d380b158155cc2c4","entry":"DAM","repo":"jayroxis/dam-pytorch","repo_kind":"official","path":"Network Pruning/models/dam.py","file_url":"https://github.com/jayroxis/dam-pytorch/blob/HEAD/Network%20Pruning/models/dam.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d380b158155cc2c4"}},{"code_sha256_prefix":"9da87b0b1605a660","entry":"num_graphs","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"9da87b0b1605a660"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}