{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-28743","title":"Rethinking Language Model Scaling under Transferable Hypersphere Optimization","arxiv_id":"2603.28743","date":"2026-03-30","proceeding":null,"authors":["Liliang Ren","Yang Liu","Yelong Shen","Weizhu Chen"],"abstract":"Scaling laws for large language models depend critically on the optimizer and parameterization. Existing hyperparameter transfer laws are mainly developed for first-order optimizers, and they do not structurally prevent training instability at scale. Recent hypersphere optimization methods constrain weight matrices to a fixed-norm hypersphere, offering a promising alternative for more stable scaling. We introduce HyperP (Hypersphere Parameterization), the first framework for transferring optimal learning rates across model width, depth, training tokens, and Mixture-of-Experts (MoE) granularity under the Frobenius-sphere constraint with the Muon optimizer. We prove that weight decay is a first-order no-op on the Frobenius sphere, show that Depth-$μ$P remains necessary, and find that the optimal learning rate follows the same data-scaling power law with the \"magic exponent\" 0.32 previously observed for AdamW. A single base learning rate tuned at the smallest scale transfers across all compute budgets under HyperP, yielding $1.58\\times$ compute efficiency over a strong Muon baseline at $6\\times10^{21}$ FLOPs. Moreover, HyperP delivers transferable stability: all monitored instability indicators, including $Z$-values, output RMS, and activation outliers, remain bounded and non-increasing under training FLOPs scaling. We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities for improved granularity scaling, and show that hypersphere optimization enables substantially larger auxiliary load-balancing weights, yielding both strong performance and good expert balance. We release our training codebase at https://github.com/microsoft/ArchScale.","url_abs":"https://arxiv.org/abs/2603.28743","url_pdf":"https://arxiv.org/pdf/2603.28743","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.28743","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.28743"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/microsoft/ArchScale","reach":null}],"summary":{"ran_draft_wrong":2,"ran_fixture":1,"ran_honours":1,"unverified":16},"by_repo_kind":{"found_in_text":{"samples":20,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bc4bddefd7540078","entry":"elu_p1","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"lit_gpt/delta_net.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/lit_gpt/delta_net.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc4bddefd7540078"}},{"code_sha256_prefix":"1143997b3acc55c1","entry":"eval_index","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"lit_gpt/layer_configurator.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/lit_gpt/layer_configurator.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1143997b3acc55c1"}},{"code_sha256_prefix":"51ee91de5e3fdcaa","entry":"iceildiv","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"eval_phonebook.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/eval_phonebook.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51ee91de5e3fdcaa"}},{"code_sha256_prefix":"e544fdb6373b9f42","entry":"sum_norm","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"lit_gpt/delta_net.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/lit_gpt/delta_net.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e544fdb6373b9f42"}},{"code_sha256_prefix":"e3fda579939ce066","entry":"activation_budget_bce","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"lit_gpt/aux_loss.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/lit_gpt/aux_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3fda579939ce066"}},{"code_sha256_prefix":"e2e4134ae510653b","entry":"calculate_flops","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"plot_flops_scaling.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/plot_flops_scaling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2e4134ae510653b"}},{"code_sha256_prefix":"51a37b13218b6913","entry":"compress_seq","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"lit_gpt/layer_configurator.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/lit_gpt/layer_configurator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51a37b13218b6913"}},{"code_sha256_prefix":"9c8d36bdd9723294","entry":"compute_rmsnorm","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"lit_gpt/model.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/lit_gpt/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c8d36bdd9723294"}},{"code_sha256_prefix":"bdd0edf8ef9c1c49","entry":"compute_tokens_per_expert_seq","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"lit_gpt/aux_loss.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/lit_gpt/aux_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdd0edf8ef9c1c49"}},{"code_sha256_prefix":"3b3d3416a93b6405","entry":"extract","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"lit_gpt/layer_configurator.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/lit_gpt/layer_configurator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b3d3416a93b6405"}},{"code_sha256_prefix":"4a60331c60ddcae3","entry":"fit_optimal_loss","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"plots/plot_loss_npoints_sensitivity.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/plots/plot_loss_npoints_sensitivity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a60331c60ddcae3"}},{"code_sha256_prefix":"6ac53a7ef2435384","entry":"get_grad_norm","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"logging_utils.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/logging_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ac53a7ef2435384"}},{"code_sha256_prefix":"af6ecca19cb81a1c","entry":"load_balance_loss","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"lit_gpt/aux_loss.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/lit_gpt/aux_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af6ecca19cb81a1c"}},{"code_sha256_prefix":"0df6477d542f5cd7","entry":"maxvio_batch","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"logging_utils.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/logging_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0df6477d542f5cd7"}},{"code_sha256_prefix":"79e61238be70ea1e","entry":"parse_name","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"plots/plot_loss_npoints_sensitivity.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/plots/plot_loss_npoints_sensitivity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79e61238be70ea1e"}},{"code_sha256_prefix":"88dcd7b69208a798","entry":"power_law","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"plots/plot_loss_npoints_sensitivity.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/plots/plot_loss_npoints_sensitivity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88dcd7b69208a798"}},{"code_sha256_prefix":"da1aea3415038667","entry":"power_law_with_offset","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"plot_data_scaling.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/plot_data_scaling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da1aea3415038667"}},{"code_sha256_prefix":"4dc7add0be5af66a","entry":"register_logging_hook","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"logging_utils.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/logging_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4dc7add0be5af66a"}},{"code_sha256_prefix":"66273c2be13e01a5","entry":"simple_norm","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"lit_gpt/delta_net.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/lit_gpt/delta_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66273c2be13e01a5"}},{"code_sha256_prefix":"1e4e234055203901","entry":"truncated_normal_","repo":"microsoft/ArchScale","repo_kind":"found_in_text","path":"lit_gpt/model.py","file_url":"https://github.com/microsoft/ArchScale/blob/HEAD/lit_gpt/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e4e234055203901"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}