{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-22297","title":"One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs","arxiv_id":"2605.22297","date":"2026-05-21","proceeding":null,"authors":["Di He","Songjun Tu","Keyu Wang","Lu Yin","Shiwei Liu"],"abstract":"Learning rate configuration is a fundamental aspect of modern deep learning. The prevailing practice of applying a uniform learning rate across all layers overlooks the structural heterogeneity of Transformers, potentially limiting their effectiveness as the backbone of Large Language Models (LLMs). In this paper, we introduce Layerwise Learning Rate (LLR), an adaptive scheme that assigns distinct learning rates to individual Transformer layers. Our method is grounded in Heavy-Tailed Self-Regularization (HT-SR) theory, which characterizes the empirical spectral density (ESD) of weight correlation matrices to quantify heavy-tailedness. Layers with weaker heavy-tailedness are assigned larger learning rates to accelerate training, while layers with stronger heavy-tailedness receive smaller learning rates. By tailoring learning rates in this manner, LLR promotes more balanced training across layers, leading to faster convergence and improved generalization. Extensive experiments across architectures ranging from LLaMA to GPT-nano, optimizers including AdamW and Muon, and model scales from 60M to 3B parameters with up to 100B training tokens demonstrate the effectiveness of LLR. LLR achieves up to 1.5x training speedup and consistently outperforms uniform-learning-rate baselines. In particular, it improves the average zero-shot accuracy of 1B models from 47.09% to 49.02%, and that of 3B models from 48.58% to 50.61%. A key advantage of LLR is its low tuning overhead: it can transfer nearly optimal learning-rate settings directly from the uniform baseline. Code is available at https://github.com/hed-ucas/Layer-wise-Learning-Rate.","url_abs":"https://arxiv.org/abs/2605.22297","url_pdf":"https://arxiv.org/pdf/2605.22297","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2605.22297","atlas_url":"https://app.syntology.ai/?focus=2605.22297","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.22297"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/hed-ucas/Layer-wise-Learning-Rate","reach":null}],"summary":{"ran":5,"ran_fixture":1,"ran_draft_wrong":3,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"4ed3d49ae02fab2d","entry":"adam_update","repo":"hed-ucas/Layer-wise-Learning-Rate","repo_kind":"found_in_text","path":"galore_utils/muon.py","file_url":"https://github.com/hed-ucas/Layer-wise-Learning-Rate/blob/HEAD/galore_utils/muon.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ed3d49ae02fab2d"}},{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"hed-ucas/Layer-wise-Learning-Rate","repo_kind":"found_in_text","path":"galore_utils/modeling_llama.py","file_url":"https://github.com/hed-ucas/Layer-wise-Learning-Rate/blob/HEAD/galore_utils/modeling_llama.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"f9abf6d74699cc1c","entry":"causal_attention_bias","repo":"hed-ucas/Layer-wise-Learning-Rate","repo_kind":"found_in_text","path":"olmo/model.py","file_url":"https://github.com/hed-ucas/Layer-wise-Learning-Rate/blob/HEAD/olmo/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9abf6d74699cc1c"}},{"code_sha256_prefix":"f58a7fbb020fe7cd","entry":"get_cyclical_cosine_schedule_with_min_lr","repo":"hed-ucas/Layer-wise-Learning-Rate","repo_kind":"found_in_text","path":"galore_utils/training_utils.py","file_url":"https://github.com/hed-ucas/Layer-wise-Learning-Rate/blob/HEAD/galore_utils/training_utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f58a7fbb020fe7cd"}},{"code_sha256_prefix":"ccb36014e20ca2ae","entry":"get_linear_schedule_to_target_lrs","repo":"hed-ucas/Layer-wise-Learning-Rate","repo_kind":"found_in_text","path":"galore_utils/training_utils.py","file_url":"https://github.com/hed-ucas/Layer-wise-Learning-Rate/blob/HEAD/galore_utils/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ccb36014e20ca2ae"}},{"code_sha256_prefix":"1e05670874fea1ef","entry":"longest_common_prefix","repo":"hed-ucas/Layer-wise-Learning-Rate","repo_kind":"found_in_text","path":"hf_olmo/convert_olmo_to_hf.py","file_url":"https://github.com/hed-ucas/Layer-wise-Learning-Rate/blob/HEAD/hf_olmo/convert_olmo_to_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e05670874fea1ef"}},{"code_sha256_prefix":"a21a1aeb7c4ef8ed","entry":"muon_update","repo":"hed-ucas/Layer-wise-Learning-Rate","repo_kind":"found_in_text","path":"galore_utils/muon.py","file_url":"https://github.com/hed-ucas/Layer-wise-Learning-Rate/blob/HEAD/galore_utils/muon.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a21a1aeb7c4ef8ed"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"hed-ucas/Layer-wise-Learning-Rate","repo_kind":"found_in_text","path":"galore_utils/modeling_llama.py","file_url":"https://github.com/hed-ucas/Layer-wise-Learning-Rate/blob/HEAD/galore_utils/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"88b19bfaad9384f8","entry":"zeropower_via_newtonschulz5","repo":"hed-ucas/Layer-wise-Learning-Rate","repo_kind":"found_in_text","path":"galore_utils/muon.py","file_url":"https://github.com/hed-ucas/Layer-wise-Learning-Rate/blob/HEAD/galore_utils/muon.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"88b19bfaad9384f8"}},{"code_sha256_prefix":"ef19a575bfdbcc82","entry":"get_scheculer","repo":"hed-ucas/Layer-wise-Learning-Rate","repo_kind":"found_in_text","path":"galore_utils/training_utils.py","file_url":"https://github.com/hed-ucas/Layer-wise-Learning-Rate/blob/HEAD/galore_utils/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef19a575bfdbcc82"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":1,"rejected_by_independent_check":0,"refused_by_a_rule":1,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}