{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scalable-efficient-training-of-large-language","title":"Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention","arxiv_id":"2411.02063","date":"2024-11-04","proceeding":null,"authors":["Xingtai Lv","Ning Ding","Kaiyan Zhang","Ermo Hua","Ganqu Cui","BoWen Zhou"],"abstract":"Improving the effectiveness and efficiency of large language models (LLMs) simultaneously is a critical yet challenging research goal. In this paper, we find that low-rank pre-training, normally considered as efficient methods that will compromise performance, can be scalably effective when reduced parameters are precisely targeted. Specifically, applying the low-dimensional module only to the attention layer -- resolves this issue and enhances both effectiveness and efficiency. We refer to this structure as Low-dimensional Projected Attention (LPA) and provide an explanatory analysis. Through extensive experimentation at parameter scales of 130M, 370M, and scaling up to 3B, we have validated the effectiveness and scalability of LPA. Our results show that LPA model can save up to 12.4% in time while achieving an approximate 5% improvement in test perplexity (ppl) and on downstream tasks compared with the vanilla Transformer.","url_abs":"https://arxiv.org/abs/2411.02063v1","url_pdf":"https://arxiv.org/pdf/2411.02063v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scalable-efficient-training-of-large-language","repo_url":"https://github.com/tsinghuac3i/lpa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2411.02063","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.02063"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tsinghuac3i/lpa","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/TsinghuaC3I/LPA","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_fixture":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"89d16230dc1ce5b0","entry":"LPA","repo":"TsinghuaC3I/LPA","repo_kind":"official","path":"architecture/lpa_setting1.py","file_url":"https://github.com/TsinghuaC3I/LPA/blob/HEAD/architecture/lpa_setting1.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89d16230dc1ce5b0"}},{"code_sha256_prefix":"8b97b03d680f6206","entry":"LPALayer","repo":"TsinghuaC3I/LPA","repo_kind":"official","path":"architecture/lpa_setting1.py","file_url":"https://github.com/TsinghuaC3I/LPA/blob/HEAD/architecture/lpa_setting1.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b97b03d680f6206"}},{"code_sha256_prefix":"fa104895b67665ee","entry":"MultiHeadAttention","repo":"TsinghuaC3I/LPA","repo_kind":"official","path":"architecture/lpa_setting1.py","file_url":"https://github.com/TsinghuaC3I/LPA/blob/HEAD/architecture/lpa_setting1.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa104895b67665ee"}},{"code_sha256_prefix":"9a65a30d006fc96e","entry":"apply_rotary_pos_emb","repo":"tsinghuac3i/lpa","repo_kind":"official","path":"architecture/lpa_setting2.py","file_url":"https://github.com/tsinghuac3i/lpa/blob/HEAD/architecture/lpa_setting2.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a65a30d006fc96e"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"tsinghuac3i/lpa","repo_kind":"official","path":"architecture/lpa_setting2.py","file_url":"https://github.com/tsinghuac3i/lpa/blob/HEAD/architecture/lpa_setting2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}