{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/don-t-be-lazy-completep-enables-compute","title":"Don't be lazy: CompleteP enables compute-efficient deep transformers","arxiv_id":"2505.01618","date":"2025-05-02","proceeding":null,"authors":["Nolan Dey","Bin Claire Zhang","Lorenzo Noci","Mufan Li","Blake Bordelon","Shane Bergsma","Cengiz Pehlevan","Boris Hanin","Joel Hestness"],"abstract":"We study compute efficiency of LLM training when using different parameterizations, i.e., rules for adjusting model and optimizer hyperparameters (HPs) as model size changes. Some parameterizations fail to transfer optimal base HPs (such as learning rate) across changes in model depth, requiring practitioners to either re-tune these HPs as they scale up (expensive), or accept sub-optimal training when re-tuning is prohibitive. Even when they achieve HP transfer, we develop theory to show parameterizations may still exist in the lazy learning regime where layers learn only features close to their linearization, preventing effective use of depth and nonlinearity. Finally, we identify and adopt the parameterization we call CompleteP that achieves both depth-wise HP transfer and non-lazy learning in all layers. CompleteP enables a wider range of model width/depth ratios to remain compute-efficient, unlocking shapes better suited for different hardware settings and operational contexts. Moreover, CompleteP enables 12-34% compute efficiency improvements over the prior state-of-the-art.","url_abs":"https://arxiv.org/abs/2505.01618v2","url_pdf":"https://arxiv.org/pdf/2505.01618v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"don-t-be-lazy-completep-enables-compute","repo_url":"https://github.com/eleutherai/nanogpt-mup","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"adopt","method_name":"ADOPT"},{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.01618","atlas_url":"https://app.syntology.ai/?focus=2505.01618","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.01618"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/EleutherAI/nanoGPT-mup","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eleutherai/nanogpt-mup","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"608e364a9d2376a3","entry":"exists","repo":"EleutherAI/nanoGPT-mup","repo_kind":"official","path":"csv_logging.py","file_url":"https://github.com/EleutherAI/nanoGPT-mup/blob/HEAD/csv_logging.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"608e364a9d2376a3"}},{"code_sha256_prefix":"3e102574d29e6b9b","entry":"transform_format_string","repo":"EleutherAI/nanoGPT-mup","repo_kind":"official","path":"csv_logging.py","file_url":"https://github.com/EleutherAI/nanoGPT-mup/blob/HEAD/csv_logging.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e102574d29e6b9b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}