{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-06615","title":"When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\\ell_1$-norm Lower Bounds","arxiv_id":"2605.06615","date":"2026-05-07","proceeding":null,"authors":["Hongyi Tao","Dingzhi Yu","Lijun Zhang"],"abstract":"Sign-based optimization algorithms, such as SignSGD and Muon, have garnered significant attention for their remarkable performance in training large foundation models. Despite this empirical success, we still lack a theoretical understanding of when and why these sign-based methods outperform vanilla SGD. The core obstacle is that under standard smoothness and finite variance conditions, SGD is known to be minimax optimal for finding stationary points measured by $\\ell_2$-norms, thereby fundamentally precluding any complexity gains for sign-based methods in standard settings. To overcome this barrier, we analyze sign-based optimizers leveraging $\\ell_1$-norm stationarity, $\\ell_\\infty$-smoothness, and a separable noise model, which can better capture the coordinate-wise nature of signed updates. Under this distinct problem geometry, we derive matched upper and lower bounds for SignSGD and explicitly characterize the problem class in which SignSGD provably dominates SGD. Specifically, we compare the \\emph{upper bound of SignSGD} with the \\emph{lower bound of SGD}, illustrating that SignSGD effectively reduces the complexity by a factor of $d$ under \\emph{sparse noise}, where $d$ is the problem dimension. Furthermore, we elevate this framework to the matrix domain, providing an equivalent optimal lower bound for the Muon optimizer, proving that extending the sign operator to matrices preserves this optimal scaling with dimensionality. Finally, we bridge our theoretical bounds to practice, demonstrating that the theoretical superiority of SignSGD accurately predicts its faster convergence during the pretraining of a 124M parameter GPT-2 model.","url_abs":"https://arxiv.org/abs/2605.06615","url_pdf":"https://arxiv.org/pdf/2605.06615","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.06615","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.06615"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Dingzhen230/SignSGD_Outperforms_SGD","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/epfml/llm-optimizer-benchmark","reach":null}],"summary":{"ran":8,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":12,"ran":8,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"1f1b81b26e97e55b","entry":"entropy_reg","repo":"epfml/llm-optimizer-benchmark","repo_kind":"found_in_text","path":"src/models/moe.py","file_url":"https://github.com/epfml/llm-optimizer-benchmark/blob/HEAD/src/models/moe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f1b81b26e97e55b"}},{"code_sha256_prefix":"a69212d334eb9bc7","entry":"load_balancing_loss_","repo":"epfml/llm-optimizer-benchmark","repo_kind":"found_in_text","path":"src/models/moe.py","file_url":"https://github.com/epfml/llm-optimizer-benchmark/blob/HEAD/src/models/moe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a69212d334eb9bc7"}},{"code_sha256_prefix":"2f520af81f2675ca","entry":"log_mean","repo":"epfml/llm-optimizer-benchmark","repo_kind":"found_in_text","path":"src/models/moe.py","file_url":"https://github.com/epfml/llm-optimizer-benchmark/blob/HEAD/src/models/moe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f520af81f2675ca"}},{"code_sha256_prefix":"1bd86a911c3f78a3","entry":"move_to_cpu","repo":"epfml/llm-optimizer-benchmark","repo_kind":"found_in_text","path":"src/logger/logger.py","file_url":"https://github.com/epfml/llm-optimizer-benchmark/blob/HEAD/src/logger/logger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1bd86a911c3f78a3"}},{"code_sha256_prefix":"a1d6f89d43fc42e1","entry":"precompute_freqs_cis","repo":"epfml/llm-optimizer-benchmark","repo_kind":"found_in_text","path":"src/models/llama.py","file_url":"https://github.com/epfml/llm-optimizer-benchmark/blob/HEAD/src/models/llama.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1d6f89d43fc42e1"}},{"code_sha256_prefix":"d83a5bca8e23539e","entry":"run_experiment","repo":"Dingzhen230/SignSGD_Outperforms_SGD","repo_kind":"found_in_text","path":"toy_models/sto.py","file_url":"https://github.com/Dingzhen230/SignSGD_Outperforms_SGD/blob/HEAD/toy_models/sto.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d83a5bca8e23539e"}},{"code_sha256_prefix":"2c4b2d77d356e906","entry":"run_trajectory","repo":"Dingzhen230/SignSGD_Outperforms_SGD","repo_kind":"found_in_text","path":"toy_models/deter.py","file_url":"https://github.com/Dingzhen230/SignSGD_Outperforms_SGD/blob/HEAD/toy_models/deter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c4b2d77d356e906"}},{"code_sha256_prefix":"198aea734522a443","entry":"self_preserving_overwrite","repo":"epfml/llm-optimizer-benchmark","repo_kind":"found_in_text","path":"src/logger/logger.py","file_url":"https://github.com/epfml/llm-optimizer-benchmark/blob/HEAD/src/logger/logger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"198aea734522a443"}},{"code_sha256_prefix":"9891174b43bd9c9a","entry":"apply_rotary_emb","repo":"epfml/llm-optimizer-benchmark","repo_kind":"found_in_text","path":"src/models/llama.py","file_url":"https://github.com/epfml/llm-optimizer-benchmark/blob/HEAD/src/models/llama.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9891174b43bd9c9a"}},{"code_sha256_prefix":"d2ddaef321aaffbf","entry":"get_model","repo":"epfml/llm-optimizer-benchmark","repo_kind":"found_in_text","path":"src/models/utils.py","file_url":"https://github.com/epfml/llm-optimizer-benchmark/blob/HEAD/src/models/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d2ddaef321aaffbf"}},{"code_sha256_prefix":"2fc6fdb85f13dc41","entry":"none_or_str","repo":"Dingzhen230/SignSGD_Outperforms_SGD","repo_kind":"found_in_text","path":"nanoGPT_expt/src/config/base.py","file_url":"https://github.com/Dingzhen230/SignSGD_Outperforms_SGD/blob/HEAD/nanoGPT_expt/src/config/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fc6fdb85f13dc41"}},{"code_sha256_prefix":"14221e01e48f4caf","entry":"sample_cifar_gradient","repo":"Dingzhen230/SignSGD_Outperforms_SGD","repo_kind":"found_in_text","path":"resnet_expt/gradient_utils.py","file_url":"https://github.com/Dingzhen230/SignSGD_Outperforms_SGD/blob/HEAD/resnet_expt/gradient_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14221e01e48f4caf"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}