{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-18106","title":"Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers","arxiv_id":"2605.18106","date":"2026-05-18","proceeding":null,"authors":["Tim Tsz-Kit Lau","Weijie Su"],"abstract":"A striking geometric disparity has long persisted in the practice of deep learning. While modern neural network architectures naturally exhibit rich symmetry and equivariance properties, popular optimizers such as Adam and its variants operate inherently coordinate-wise, rendering them unable to respect the equivariance structures of the parameter space. We address this disparity by introducing a symmetry-compatible principle for optimizer design: the gradient update rule should be equivariant under the symmetry group acting on the corresponding weight block. Following this principle, we first provide a unified perspective on bi-orthogonally equivariant updates for general matrix layers, as employed by stochastic spectral descent, Muon, Scion, and polar gradient methods. More importantly, by moving from orthogonal groups to permutation and shared-shift symmetries, we derive symmetry-compatible optimizers for parameter blocks whose symmetries differ from those of general matrix layers: embedding and LM head matrices, SwiGLU MLP projections, and MoE router matrices. These constructions include one-sided spectral, row-norm, hybrid row-norm/spectral, row-aware, column-aware, centered row-norm, and left-spectral updates. They yield an end-to-end layerwise optimizer stack in which each major matrix-valued parameter class is assigned an update whose equivariance matches its symmetry group. We corroborate this principle through pre-training experiments on dense and sparse MoE language models, including Qwen3-0.6B-style, Gemma 3 1B-style, OLMoE-1B-7B-style, and downsized gpt-oss architectures. Across these experiments, symmetry-compatible update rules consistently improve final validation loss, reduce expert load imbalance in sparse MoE models, and in several cases control final vocabulary-logit growth, improve router stability, and overall training stability over the corresponding AdamW updates.","url_abs":"https://arxiv.org/abs/2605.18106","url_pdf":"https://arxiv.org/pdf/2605.18106","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2605.18106","atlas_url":"https://app.syntology.ai/?focus=2605.18106","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.18106"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/timlautk/equivariant_optimizers","reach":null}],"summary":{"ran":1,"ran_violates":2,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bb62e664b5e49b4d","entry":"LeftPolarGradM","repo":"timlautk/equivariant_optimizers","repo_kind":"found_in_text","path":"optim/leftpolargrad.py","file_url":"https://github.com/timlautk/equivariant_optimizers/blob/HEAD/optim/leftpolargrad.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb62e664b5e49b4d"}},{"code_sha256_prefix":"6cc1a186129c5d0e","entry":"is_matrix_param","repo":"timlautk/equivariant_optimizers","repo_kind":"found_in_text","path":"optim/leftpolargrad.py","file_url":"https://github.com/timlautk/equivariant_optimizers/blob/HEAD/optim/leftpolargrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6cc1a186129c5d0e"}},{"code_sha256_prefix":"3fba78841d11b568","entry":"symmetrize","repo":"timlautk/equivariant_optimizers","repo_kind":"found_in_text","path":"optim/leftpolargrad.py","file_url":"https://github.com/timlautk/equivariant_optimizers/blob/HEAD/optim/leftpolargrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3fba78841d11b568"}},{"code_sha256_prefix":"3609593b5035e8d4","entry":"decoupled_weight_decay_","repo":"timlautk/equivariant_optimizers","repo_kind":"found_in_text","path":"optim/leftpolargrad.py","file_url":"https://github.com/timlautk/equivariant_optimizers/blob/HEAD/optim/leftpolargrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3609593b5035e8d4"}},{"code_sha256_prefix":"178af75ac58ca1ca","entry":"symmetric_matrix_invsqrt","repo":"timlautk/equivariant_optimizers","repo_kind":"found_in_text","path":"optim/leftpolargrad.py","file_url":"https://github.com/timlautk/equivariant_optimizers/blob/HEAD/optim/leftpolargrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"178af75ac58ca1ca"}},{"code_sha256_prefix":"8c6c55560efc5af8","entry":"symmetric_matrix_invsqrt_newton_schulz","repo":"timlautk/equivariant_optimizers","repo_kind":"found_in_text","path":"optim/leftpolargrad.py","file_url":"https://github.com/timlautk/equivariant_optimizers/blob/HEAD/optim/leftpolargrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c6c55560efc5af8"}},{"code_sha256_prefix":"f8fd62b24d872815","entry":"symmetric_matrix_invsqrt_polar_express","repo":"timlautk/equivariant_optimizers","repo_kind":"found_in_text","path":"optim/leftpolargrad.py","file_url":"https://github.com/timlautk/equivariant_optimizers/blob/HEAD/optim/leftpolargrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f8fd62b24d872815"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}