{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/supermasks-in-superposition","title":"Supermasks in Superposition","arxiv_id":"2006.14769","date":"2020-06-26","proceeding":"NeurIPS 2020 12","authors":["Mitchell Wortsman","Vivek Ramanujan","Rosanne Liu","Aniruddha Kembhavi","Mohammad Rastegari","Jason Yosinski","Ali Farhadi"],"abstract":"We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting. Our approach uses a randomly initialized, fixed base network and for each task finds a subnetwork (supermask) that achieves good performance. If task identity is given at test time, the correct subnetwork can be retrieved with minimal memory usage. If not provided, SupSup can infer the task using gradient-based optimization to find a linear superposition of learned supermasks which minimizes the output entropy. In practice we find that a single gradient step is often sufficient to identify the correct mask, even among 2500 tasks. We also showcase two promising extensions. First, SupSup models can be trained entirely without task identity information, as they may detect when they are uncertain about new data and allocate an additional supermask for the new training distribution. Finally the entire, growing set of supermasks can be stored in a constant-sized reservoir by implicitly storing them as attractors in a fixed-sized Hopfield network.","url_abs":"https://arxiv.org/abs/2006.14769v3","url_pdf":"https://arxiv.org/pdf/2006.14769v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"supermasks-in-superposition","repo_url":"https://github.com/RAIVNLab/supsup","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"supermasks-in-superposition","repo_url":"https://github.com/dmiu-shell/deeprl-shell","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.14769","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.14769"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RAIVNLab/supsup","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dmiu-shell/deeprl-shell","reach":null}],"summary":{"ran":3,"ran_draft_wrong":1,"ran_honours":2,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1},"listed":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"322b181b10018053","entry":"GetSubnetContinuous","repo":"dmiu-shell/deeprl-shell","repo_kind":"listed","path":"deep_rl/shell_modules/mmn/ssmask_utils.py","file_url":"https://github.com/dmiu-shell/deeprl-shell/blob/HEAD/deep_rl/shell_modules/mmn/ssmask_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"322b181b10018053"}},{"code_sha256_prefix":"4de2d75cb07fd186","entry":"GetSubnetDiscrete","repo":"dmiu-shell/deeprl-shell","repo_kind":"listed","path":"deep_rl/shell_modules/mmn/ssmask_utils.py","file_url":"https://github.com/dmiu-shell/deeprl-shell/blob/HEAD/deep_rl/shell_modules/mmn/ssmask_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4de2d75cb07fd186"}},{"code_sha256_prefix":"31da33c140f3a9c8","entry":"MultitaskMaskLinear","repo":"dmiu-shell/deeprl-shell","repo_kind":"listed","path":"deep_rl/shell_modules/mmn/ssmask_utils.py","file_url":"https://github.com/dmiu-shell/deeprl-shell/blob/HEAD/deep_rl/shell_modules/mmn/ssmask_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"31da33c140f3a9c8"}},{"code_sha256_prefix":"a407b1a72b42c61f","entry":"get_subnet","repo":"RAIVNLab/supsup","repo_kind":"official","path":"models/modules.py","file_url":"https://github.com/RAIVNLab/supsup/blob/HEAD/models/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a407b1a72b42c61f"}},{"code_sha256_prefix":"ffe66fb2efd3fe30","entry":"mask_init","repo":"RAIVNLab/supsup","repo_kind":"official","path":"models/modules.py","file_url":"https://github.com/RAIVNLab/supsup/blob/HEAD/models/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ffe66fb2efd3fe30"}},{"code_sha256_prefix":"4041eef325f5a3b2","entry":"mask_init","repo":"dmiu-shell/deeprl-shell","repo_kind":"listed","path":"deep_rl/shell_modules/mmn/ssmask_utils.py","file_url":"https://github.com/dmiu-shell/deeprl-shell/blob/HEAD/deep_rl/shell_modules/mmn/ssmask_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4041eef325f5a3b2"}},{"code_sha256_prefix":"1d23e73da3824929","entry":"MultitaskMaskConv","repo":"RAIVNLab/supsup","repo_kind":"official","path":"models/modules.py","file_url":"https://github.com/RAIVNLab/supsup/blob/HEAD/models/modules.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d23e73da3824929"}},{"code_sha256_prefix":"f2849754778de9a4","entry":"kwargs_to_cmd","repo":"RAIVNLab/supsup","repo_kind":"official","path":"experiments/GG/splitcifar100/rn18-supsup.py","file_url":"https://github.com/RAIVNLab/supsup/blob/HEAD/experiments/GG/splitcifar100/rn18-supsup.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2849754778de9a4"}},{"code_sha256_prefix":"3220897509d99823","entry":"signed_constant","repo":"dmiu-shell/deeprl-shell","repo_kind":"listed","path":"deep_rl/shell_modules/mmn/ssmask_utils.py","file_url":"https://github.com/dmiu-shell/deeprl-shell/blob/HEAD/deep_rl/shell_modules/mmn/ssmask_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3220897509d99823"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}