{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/non-proportional-parametrizations-for-stable","title":"Magnitude Invariant Parametrizations Improve Hypernetwork Learning","arxiv_id":"2304.07645","date":"2023-04-15","proceeding":null,"authors":["Jose Javier Gonzalez Ortiz","John Guttag","Adrian Dalca"],"abstract":"Hypernetworks, neural networks that predict the parameters of another neural network, are powerful models that have been successfully used in diverse applications from image generation to multi-task learning. Unfortunately, existing hypernetworks are often challenging to train. Training typically converges far more slowly than for non-hypernetwork models, and the rate of convergence can be very sensitive to hyperparameter choices. In this work, we identify a fundamental and previously unidentified problem that contributes to the challenge of training hypernetworks: a magnitude proportionality between the inputs and outputs of the hypernetwork. We demonstrate both analytically and empirically that this can lead to unstable optimization, thereby slowing down convergence, and sometimes even preventing any learning. We present a simple solution to this problem using a revised hypernetwork formulation that we call Magnitude Invariant Parametrizations (MIP). We demonstrate the proposed solution on several hypernetwork tasks, where it consistently stabilizes training and achieves faster convergence. Furthermore, we perform a comprehensive ablation study including choices of activation function, normalization strategies, input dimensionality, and hypernetwork architecture; and find that MIP improves training in all scenarios. We provide easy-to-use code that can turn existing networks into MIP-based hypernetworks.","url_abs":"https://arxiv.org/abs/2304.07645v2","url_pdf":"https://arxiv.org/pdf/2304.07645v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"non-proportional-parametrizations-for-stable","repo_url":"https://github.com/jjgo/hyperlight","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"}],"methods":[{"method_slug":"hypernetwork","method_name":"HyperNetwork"},{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.07645","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.07645"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jjgo/hyperlight","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"97c41a7d954d5827","entry":"encode_input","repo":"jjgo/hyperlight","repo_kind":"official","path":"hyperlight/hypernet/encoding.py","file_url":"https://github.com/jjgo/hyperlight/blob/HEAD/hyperlight/hypernet/encoding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"97c41a7d954d5827"}},{"code_sha256_prefix":"aa22a8590ebf26ed","entry":"find_modules_from_patterns","repo":"jjgo/hyperlight","repo_kind":"official","path":"hyperlight/find.py","file_url":"https://github.com/jjgo/hyperlight/blob/HEAD/hyperlight/find.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa22a8590ebf26ed"}},{"code_sha256_prefix":"a7f1cddd4723539a","entry":"find_modules_of_type","repo":"jjgo/hyperlight","repo_kind":"official","path":"hyperlight/find.py","file_url":"https://github.com/jjgo/hyperlight/blob/HEAD/hyperlight/find.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7f1cddd4723539a"}},{"code_sha256_prefix":"141977438168693a","entry":"find_parameters_from_patterns","repo":"jjgo/hyperlight","repo_kind":"official","path":"hyperlight/find.py","file_url":"https://github.com/jjgo/hyperlight/blob/HEAD/hyperlight/find.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"141977438168693a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}