{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/comera-computing-and-memory-efficient","title":"CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization","arxiv_id":"2405.14377","date":"2024-05-23","proceeding":null,"authors":["Zi Yang","Ziyue Liu","Samridhi Choudhary","Xinfeng Xie","Cao Gao","Siegfried Kunzmann","Zheng Zhang"],"abstract":"Training large AI models such as LLMs and DLRMs costs massive GPUs and computing time. The high training cost has become only affordable to big tech companies, meanwhile also causing increasing concerns about the environmental impact. This paper presents CoMERA, a Computing- and Memory-Efficient training method via Rank-Adaptive tensor optimization. CoMERA achieves rank-adaptive tensor-compressed (pre)-training via a multi-objective optimization formulation and improves the training to provide both a high compression ratio and excellent accuracy in the training process. Our optimized numerical computation (e.g., optimized tensorized embedding and tensor-network contractions) and GPU implementation eliminate part of the run-time overhead in the tensorized training on GPU. This leads to, for the first time, $2-3\\times$ speedup per training epoch compared with standard training. CoMERA also outperforms the recent GaLore in terms of both memory and computing efficiency. Specifically, CoMERA is $2\\times$ faster per training epoch and $9\\times$ more memory-efficient than GaLore on a tested six-encoder transformer with single-batch training. Our method also shows $\\sim 2\\times$ speedup than standard pre-training on a BERT-like code-generation LLM while achieving $4.23\\times$ compression ratio in pre-training. With further HPC optimization, CoMERA may reduce the pre-training cost of many other LLMs. An implementation of CoMERA is available at https://github.com/ziyangjoy/CoMERA.","url_abs":"https://arxiv.org/abs/2405.14377v2","url_pdf":"https://arxiv.org/pdf/2405.14377v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"comera-computing-and-memory-efficient","repo_url":"https://github.com/ziyangjoy/comera","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.14377","atlas_url":"https://app.syntology.ai/?focus=2405.14377","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.14377"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ziyangjoy/CoMERA","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ziyangjoy/comera","reach":{"status":"ok"}}],"summary":{"ran":6,"ran_draft_wrong":3,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"3b4208c2e336c84c","entry":"TT_forward","repo":"ziyangjoy/comera","repo_kind":"official","path":"tensor_layers_CoMERA/layers.py","file_url":"https://github.com/ziyangjoy/comera/blob/HEAD/tensor_layers_CoMERA/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b4208c2e336c84c"}},{"code_sha256_prefix":"74e4c79da1ad1e63","entry":"collate_fn_custom","repo":"ziyangjoy/comera","repo_kind":"official","path":"MNLI_trainer.py","file_url":"https://github.com/ziyangjoy/comera/blob/HEAD/MNLI_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74e4c79da1ad1e63"}},{"code_sha256_prefix":"dfbfbadf59838ca7","entry":"collate_fn_custom_graph_old","repo":"ziyangjoy/comera","repo_kind":"official","path":"MNLI_trainer.py","file_url":"https://github.com/ziyangjoy/comera/blob/HEAD/MNLI_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfbfbadf59838ca7"}},{"code_sha256_prefix":"7067bf064400d6c5","entry":"collate_fn_custom_old","repo":"ziyangjoy/comera","repo_kind":"official","path":"MNLI_trainer.py","file_url":"https://github.com/ziyangjoy/comera/blob/HEAD/MNLI_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7067bf064400d6c5"}},{"code_sha256_prefix":"f2b48295cc45f2df","entry":"config_class","repo":"ziyangjoy/comera","repo_kind":"official","path":"tensor_layers_CoMERA/layers.py","file_url":"https://github.com/ziyangjoy/comera/blob/HEAD/tensor_layers_CoMERA/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2b48295cc45f2df"}},{"code_sha256_prefix":"ea0fb1428f2cc2e8","entry":"split_galore_params","repo":"ziyangjoy/CoMERA","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ziyangjoy/CoMERA/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea0fb1428f2cc2e8"}},{"code_sha256_prefix":"aee26c824bf192b9","entry":"split_tensor_params","repo":"ziyangjoy/CoMERA","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ziyangjoy/CoMERA/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aee26c824bf192b9"}},{"code_sha256_prefix":"993569c3fe92ff48","entry":"split_tensor_rank_params","repo":"ziyangjoy/CoMERA","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ziyangjoy/CoMERA/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"993569c3fe92ff48"}},{"code_sha256_prefix":"2afb43ca8be6a440","entry":"truncate","repo":"ziyangjoy/CoMERA","repo_kind":"official","path":"data_process.py","file_url":"https://github.com/ziyangjoy/CoMERA/blob/HEAD/data_process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2afb43ca8be6a440"}},{"code_sha256_prefix":"733fa1ce8c23101d","entry":"TensorTrain","repo":"ziyangjoy/comera","repo_kind":"official","path":"tensor_layers_CoMERA/layers.py","file_url":"https://github.com/ziyangjoy/comera/blob/HEAD/tensor_layers_CoMERA/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"733fa1ce8c23101d"}},{"code_sha256_prefix":"3b58c2c59a777c56","entry":"TensorizedLinear_module","repo":"ziyangjoy/comera","repo_kind":"official","path":"tensor_layers_CoMERA/layers.py","file_url":"https://github.com/ziyangjoy/comera/blob/HEAD/tensor_layers_CoMERA/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b58c2c59a777c56"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}