{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-07436","title":"Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers","arxiv_id":"2608.07436","date":"2026-08-07","proceeding":null,"authors":["Ali Janati","Kaoutar El Maghraoui","Andrei Kanavalau","Anass Belfatmi"],"abstract":"Under the standard split, Muon gets hidden matrices and AdamW embeddings/output head. Muon groks modular addition faster, but its solutions do not hold. All nine configurations on $(a+b) \\bmod 113$ grok and later lose generalization. Across five seeds the selected AdamW reference falls below threshold on four, reaching 27.59%. Instability persists across two moduli, two widths, two training fractions, subtraction, and depth. The failure arises at the representation-readout interface, identified only jointly up to an invertible map unselected by the loss. After solving the training set, the gradient falls to order $10^{-6}$ and the optimizers respond differently: step-size elasticity is -0.03 for Muon versus +1.5 for AdamW, and the Muon group moves 8.0 times faster per parameter. From bit-identical states, freezing either group prevents failure. Freezing embeddings/readout removes it in five runs over 451,400 post-grokking steps and five paired seeds: unfrozen arms record 137-321 sub-threshold evaluations, frozen arms none. Removing Muon's normalization and orthogonalization is no substitute: it collapses representation from 326 effective conjugate pairs to 4, shows no recurrent collapse, and fails terminally. Fourier filtering separates circuit failure from masking. Across 43 checkpoints over five seeds and three regimes, the task-aligned family reaches exactly 100% alone. In circuit failure it no longer solves the task; in masking it remains perfect while the full model reaches 45.85%, giving a positive margin on every example, including errors, but being outvoted by a near-equal adversarial remainder. Rescaling it restores 99.9%; grokking is the same condition resolving upward. The task selects the family, swapping $(k,k)$ for $(k,-k)$ under subtraction. Across an abrupt collapse, standard Fourier support is unchanged and the power-distribution cosine remains 0.9899.","url_abs":"https://arxiv.org/abs/2608.07436","url_pdf":"https://arxiv.org/pdf/2608.07436","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2608.07436","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.07436"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Na00s/muon-grokking","reach":null}],"summary":{"ran":18,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":19,"ran":18,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":19,"samples":[{"code_sha256_prefix":"7088443cb52cce9b","entry":"applied_update_statistics","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"metrics.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7088443cb52cce9b"}},{"code_sha256_prefix":"1a01685cadf1a70c","entry":"checkpoint_step","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/collapse_spectral_analysis.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/collapse_spectral_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a01685cadf1a70c"}},{"code_sha256_prefix":"61c353b9219b72ad","entry":"clone_parameter_values","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"metrics.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61c353b9219b72ad"}},{"code_sha256_prefix":"6b2865ae4dfd0218","entry":"discover_checkpoints","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/collapse_spectral_analysis.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/collapse_spectral_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b2865ae4dfd0218"}},{"code_sha256_prefix":"57cdd008a4b1827b","entry":"first_step","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/sweeps/analyze_muon_sweep.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/sweeps/analyze_muon_sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57cdd008a4b1827b"}},{"code_sha256_prefix":"5328f55ec648437b","entry":"first_sustained_step","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/compare_grokking_speed.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/compare_grokking_speed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5328f55ec648437b"}},{"code_sha256_prefix":"f09f35d26223360c","entry":"first_sustained_step","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/sweeps/analyze_muon_sweep.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/sweeps/analyze_muon_sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f09f35d26223360c"}},{"code_sha256_prefix":"79e759882c0c64c4","entry":"first_threshold_step","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/compare_grokking_speed.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/compare_grokking_speed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"79e759882c0c64c4"}},{"code_sha256_prefix":"1cc60a06739b7d28","entry":"generate_modular_addition_data","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"data.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1cc60a06739b7d28"}},{"code_sha256_prefix":"2d5605e63228fd4a","entry":"get_device","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/collapse_spectral_analysis.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/collapse_spectral_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d5605e63228fd4a"}},{"code_sha256_prefix":"012c5df41664f48d","entry":"get_device","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/fourier_circuit_analysis.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/fourier_circuit_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"012c5df41664f48d"}},{"code_sha256_prefix":"f9e75d02b36ce84d","entry":"main_panel","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"experiments/depth/depth_fourier_layerwise_mechanism.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/experiments/depth/depth_fourier_layerwise_mechanism.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9e75d02b36ce84d"}},{"code_sha256_prefix":"bab898f18d19005a","entry":"matched_panel","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"experiments/depth/depth_fourier_layerwise_mechanism.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/experiments/depth/depth_fourier_layerwise_mechanism.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bab898f18d19005a"}},{"code_sha256_prefix":"da0e232dffb431c3","entry":"matched_role","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"experiments/depth/depth_fourier_layerwise_mechanism.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/experiments/depth/depth_fourier_layerwise_mechanism.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da0e232dffb431c3"}},{"code_sha256_prefix":"7986cea2cd3e4a4b","entry":"normalize_columns","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/compare_grokking_speed.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/compare_grokking_speed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7986cea2cd3e4a4b"}},{"code_sha256_prefix":"46523ebecc3f7159","entry":"parse_run_specification","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/fourier_circuit_analysis.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/fourier_circuit_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46523ebecc3f7159"}},{"code_sha256_prefix":"cad1c5142549259c","entry":"summarize_training_run","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/sweeps/analyze_muon_sweep.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/sweeps/analyze_muon_sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cad1c5142549259c"}},{"code_sha256_prefix":"b9522f666f214566","entry":"zeroth_power_newton_schulz","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"optimizers/muon.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/optimizers/muon.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b9522f666f214566"}},{"code_sha256_prefix":"d397936b6bf67685","entry":"load_checkpoint","repo":"Na00s/muon-grokking","repo_kind":"found_in_text","path":"analysis/interventions/branch_auxiliary_components.py","file_url":"https://github.com/Na00s/muon-grokking/blob/HEAD/analysis/interventions/branch_auxiliary_components.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d397936b6bf67685"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}