{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/structure-is-not-enough-leveraging-behavior","title":"Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstruction","arxiv_id":"2503.17138","date":"2025-03-21","proceeding":null,"authors":["Léo Meynent","Ivan Melev","Konstantin Schürholt","Göran Kauermann","Damian Borth"],"abstract":"The weights of neural networks (NNs) have recently gained prominence as a new data modality in machine learning, with applications ranging from accuracy and hyperparameter prediction to representation learning or weight generation. One approach to leverage NN weights involves training autoencoders (AEs), using contrastive and reconstruction losses. This allows such models to be applied to a wide variety of downstream tasks, and they demonstrate strong predictive performance and low reconstruction error. However, despite the low reconstruction error, these AEs reconstruct NN models with deteriorated performance compared to the original ones, limiting their usability with regard to model weight generation. In this paper, we identify a limitation of weight-space AEs, specifically highlighting that a structural loss, that uses the Euclidean distance between original and reconstructed weights, fails to capture some features critical for reconstructing high-performing models. We analyze the addition of a behavioral loss for training AEs in weight space, where we compare the output of the reconstructed model with that of the original one, given some common input. We show a strong synergy between structural and behavioral signals, leading to increased performance in all downstream tasks evaluated, in particular NN weights reconstruction and generation.","url_abs":"https://arxiv.org/abs/2503.17138v1","url_pdf":"https://arxiv.org/pdf/2503.17138v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"structure-is-not-enough-leveraging-behavior","repo_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.17138","atlas_url":"https://app.syntology.ai/?focus=2503.17138","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.17138"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":7},"by_repo_kind":{"official":{"samples":13,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e8143a66cdd639c4","entry":"load_checkpoint","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/dataset_base_trojai.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/dataset_base_trojai.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8143a66cdd639c4"}},{"code_sha256_prefix":"889fe8497b0ba7c8","entry":"permute_model_vector","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/augmentations.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/augmentations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"889fe8497b0ba7c8"}},{"code_sha256_prefix":"5ac745803cfc2796","entry":"read_properties_from_path","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/dataset_base.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/dataset_base.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ac745803cfc2796"}},{"code_sha256_prefix":"ca6ff6ea8664f6df","entry":"read_properties_from_path","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/dataset_base_trojai.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/dataset_base_trojai.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca6ff6ea8664f6df"}},{"code_sha256_prefix":"ec1ff9919f412754","entry":"test_checkpoint_for_nan","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/dataset_auxiliaries.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/dataset_auxiliaries.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec1ff9919f412754"}},{"code_sha256_prefix":"d2a9bd8273d8f5d4","entry":"test_checkpoint_with_threshold","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/dataset_auxiliaries.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/dataset_auxiliaries.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2a9bd8273d8f5d4"}},{"code_sha256_prefix":"6e1e8bc2c8ad772d","entry":"cast_properties_to_numbers","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/dataset_ffcv_trojai.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/dataset_ffcv_trojai.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e1e8bc2c8ad772d"}},{"code_sha256_prefix":"227a3d4f5ac6c02d","entry":"fib","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/skeleton.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/skeleton.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"227a3d4f5ac6c02d"}},{"code_sha256_prefix":"31ce1adb9a55256c","entry":"get_net_epoch_lst_from_label","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/dataset_auxiliaries.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/dataset_auxiliaries.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31ce1adb9a55256c"}},{"code_sha256_prefix":"336c00ba04cc94cf","entry":"load_checkpoint","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/dataset_base.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/dataset_base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"336c00ba04cc94cf"}},{"code_sha256_prefix":"5fb96d161e21926b","entry":"load_checkpoints_remote","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/dataset_base.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/dataset_base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fb96d161e21926b"}},{"code_sha256_prefix":"2b29b730acf17450","entry":"load_checkpoints_remote","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/dataset_base_trojai.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/dataset_base_trojai.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b29b730acf17450"}},{"code_sha256_prefix":"790ee3cda94b46c4","entry":"read_properties_from_path","repo":"HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough","repo_kind":"official","path":"skaling_hyper_reps/src/shrp/datasets/dataset_properties.py","file_url":"https://github.com/HSG-AIML/ICLR_WSL_2025-Structure_is_not_enough/blob/HEAD/skaling_hyper_reps/src/shrp/datasets/dataset_properties.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"790ee3cda94b46c4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}