{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improved-generalization-of-weight-space","title":"Improved Generalization of Weight Space Networks via Augmentations","arxiv_id":"2402.04081","date":"2024-02-06","proceeding":null,"authors":["Aviv Shamsian","Aviv Navon","David W. Zhang","Yan Zhang","Ethan Fetaya","Gal Chechik","Haggai Maron"],"abstract":"Learning in deep weight spaces (DWS), where neural networks process the weights of other neural networks, is an emerging research direction, with applications to 2D and 3D neural fields (INRs, NeRFs), as well as making inferences about other types of neural networks. Unfortunately, weight space models tend to suffer from substantial overfitting. We empirically analyze the reasons for this overfitting and find that a key reason is the lack of diversity in DWS datasets. While a given object can be represented by many different weight configurations, typical INR training sets fail to capture variability across INRs that represent the same object. To address this, we explore strategies for data augmentation in weight spaces and propose a MixUp method adapted for weight spaces. We demonstrate the effectiveness of these methods in two setups. In classification, they improve performance similarly to having up to 10 times more data. In self-supervised contrastive learning, they yield substantial 5-10% gains in downstream classification.","url_abs":"https://arxiv.org/abs/2402.04081v2","url_pdf":"https://arxiv.org/pdf/2402.04081v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improved-generalization-of-weight-space","repo_url":"https://github.com/avivsham/deep-weight-space-augmentations","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"diversity","task_name":"Diversity"}],"methods":[{"method_slug":"mixup","method_name":"Mixup"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.04081","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.04081"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/avivsham/deep-weight-space-augmentations","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":5},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"10865bbb99140edd","entry":"Linear","repo":"avivsham/deep-weight-space-augmentations","repo_kind":"official","path":"models/relation_transformer/models.py","file_url":"https://github.com/avivsham/deep-weight-space-augmentations/blob/HEAD/models/relation_transformer/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10865bbb99140edd"}},{"code_sha256_prefix":"3291d9b950f9b2b2","entry":"batch_to_graphs","repo":"avivsham/deep-weight-space-augmentations","repo_kind":"official","path":"models/relation_transformer/graph_constructor.py","file_url":"https://github.com/avivsham/deep-weight-space-augmentations/blob/HEAD/models/relation_transformer/graph_constructor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3291d9b950f9b2b2"}},{"code_sha256_prefix":"a1a1ae5b89c1989d","entry":"extract_weights_and_biases","repo":"avivsham/deep-weight-space-augmentations","repo_kind":"official","path":"augmentations/augmentations.py","file_url":"https://github.com/avivsham/deep-weight-space-augmentations/blob/HEAD/augmentations/augmentations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1a1ae5b89c1989d"}},{"code_sha256_prefix":"d1d584930ebaf49f","entry":"make_functional","repo":"avivsham/deep-weight-space-augmentations","repo_kind":"official","path":"models/relation_transformer/probe_features.py","file_url":"https://github.com/avivsham/deep-weight-space-augmentations/blob/HEAD/models/relation_transformer/probe_features.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1d584930ebaf49f"}},{"code_sha256_prefix":"1c6218d622c906cd","entry":"params_to_tensor","repo":"avivsham/deep-weight-space-augmentations","repo_kind":"official","path":"models/relation_transformer/probe_features.py","file_url":"https://github.com/avivsham/deep-weight-space-augmentations/blob/HEAD/models/relation_transformer/probe_features.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c6218d622c906cd"}},{"code_sha256_prefix":"2ca47312070f6b13","entry":"tensor_to_params","repo":"avivsham/deep-weight-space-augmentations","repo_kind":"official","path":"models/relation_transformer/probe_features.py","file_url":"https://github.com/avivsham/deep-weight-space-augmentations/blob/HEAD/models/relation_transformer/probe_features.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ca47312070f6b13"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}