{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/training-scale-invariant-neural-networks-on","title":"Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes","arxiv_id":"2209.03695","date":"2022-09-08","proceeding":null,"authors":["Maxim Kodryan","Ekaterina Lobacheva","Maksim Nakhodnov","Dmitry Vetrov"],"abstract":"A fundamental property of deep learning normalization techniques, such as batch normalization, is making the pre-normalization parameters scale invariant. The intrinsic domain of such parameters is the unit sphere, and therefore their gradient optimization dynamics can be represented via spherical optimization with varying effective learning rate (ELR), which was studied previously. However, the varying ELR may obscure certain characteristics of the intrinsic loss landscape structure. In this work, we investigate the properties of training scale-invariant neural networks directly on the sphere using a fixed ELR. We discover three regimes of such training depending on the ELR value: convergence, chaotic equilibrium, and divergence. We study these regimes in detail both on a theoretical examination of a toy example and on a thorough empirical analysis of real scale-invariant deep learning models. Each regime has unique features and reflects specific properties of the intrinsic loss landscape, some of which have strong parallels with previous research on both regular and scale-invariant neural networks training. Finally, we demonstrate how the discovered regimes are reflected in conventional training of normalized networks and how they can be leveraged to achieve better optima.","url_abs":"https://arxiv.org/abs/2209.03695v3","url_pdf":"https://arxiv.org/pdf/2209.03695v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"training-scale-invariant-neural-networks-on","repo_url":"https://github.com/tipt0p/three_regimes_on_the_sphere","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"elr","method_name":"ELR"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2209.03695","atlas_url":"https://app.syntology.ai/?focus=2209.03695","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.03695"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tipt0p/three_regimes_on_the_sphere","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/tipt0p/periodic_","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"summary":{"ran":4,"ran_draft_wrong":1,"unverified":10},"by_repo_kind":{"official":{"samples":15,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a37fca5fb609181f","entry":"block","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"nets/convnet_nobn.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/nets/convnet_nobn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a37fca5fb609181f"}},{"code_sha256_prefix":"257b3850722caed2","entry":"calc_cos","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"get_info.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/get_info.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"257b3850722caed2"}},{"code_sha256_prefix":"c67965018c758260","entry":"calc_l2","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"get_info.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/get_info.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c67965018c758260"}},{"code_sha256_prefix":"4a4f943527362329","entry":"check_si_name","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"get_info.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/get_info.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a4f943527362329"}},{"code_sha256_prefix":"6d8daee3396e21d5","entry":"unflatten_like","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"get_info_funcs.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/get_info_funcs.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6d8daee3396e21d5"}},{"code_sha256_prefix":"a164b6cd82f40a9f","entry":"ConvBNrelu","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"nets/cifar_net.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/nets/cifar_net.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a164b6cd82f40a9f"}},{"code_sha256_prefix":"19b54a137a0bbdb7","entry":"Lop","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"get_info_funcs.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/get_info_funcs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"19b54a137a0bbdb7"}},{"code_sha256_prefix":"8bc49ef2a0810046","entry":"Rop","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"get_info_funcs.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/get_info_funcs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8bc49ef2a0810046"}},{"code_sha256_prefix":"e9d3b739701f64b2","entry":"block","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"nets/convnet.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/nets/convnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e9d3b739701f64b2"}},{"code_sha256_prefix":"f9e345905349850a","entry":"block","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"nets/convnet_si.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/nets/convnet_si.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9e345905349850a"}},{"code_sha256_prefix":"bf9fa9508b358457","entry":"block","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"nets/convnet_si_af.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/nets/convnet_si_af.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bf9fa9508b358457"}},{"code_sha256_prefix":"d2c03e2aaa143601","entry":"check_si_name","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"nets/convnet_si.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/nets/convnet_si.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d2c03e2aaa143601"}},{"code_sha256_prefix":"157cf0b88320c24e","entry":"get_model_extra","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"nb_utils.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/nb_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"157cf0b88320c24e"}},{"code_sha256_prefix":"c415f0e6392ad2af","entry":"get_model_str","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"nb_utils.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/nb_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c415f0e6392ad2af"}},{"code_sha256_prefix":"182de3ac9284f86e","entry":"svhn_loaders","repo":"tipt0p/three_regimes_on_the_sphere","repo_kind":"official","path":"data.py","file_url":"https://github.com/tipt0p/three_regimes_on_the_sphere/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"182de3ac9284f86e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}