{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/collie-collaborative-training-of-large","title":"CoLLiE: Collaborative Training of Large Language Models in an Efficient Way","arxiv_id":"2312.00407","date":"2023-12-01","proceeding":null,"authors":["Kai Lv","Shuo Zhang","Tianle Gu","Shuhao Xing","Jiawei Hong","Keyu Chen","Xiaoran Liu","Yuqing Yang","Honglin Guo","Tengxiao Liu","Yu Sun","Qipeng Guo","Hang Yan","Xipeng Qiu"],"abstract":"Large language models (LLMs) are increasingly pivotal in a wide range of natural language processing tasks. Access to pre-trained models, courtesy of the open-source community, has made it possible to adapt these models to specific applications for enhanced performance. However, the substantial resources required for training these models necessitate efficient solutions. This paper introduces CoLLiE, an efficient library that facilitates collaborative training of large language models using 3D parallelism, parameter-efficient fine-tuning (PEFT) methods, and optimizers such as Lion, Adan, Sophia, LOMO and AdaLomo. With its modular design and comprehensive functionality, CoLLiE offers a balanced blend of efficiency, ease of use, and customization. CoLLiE has proven superior training efficiency in comparison with prevalent solutions in pre-training and fine-tuning scenarios. Furthermore, we provide an empirical evaluation of the correlation between model size and GPU memory consumption under different optimization methods, as well as an analysis of the throughput. Lastly, we carry out a comprehensive comparison of various optimizers and PEFT methods within the instruction-tuning context. CoLLiE is available at https://github.com/OpenLMLab/collie.","url_abs":"https://arxiv.org/abs/2312.00407v1","url_pdf":"https://arxiv.org/pdf/2312.00407v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"collie-collaborative-training-of-large","repo_url":"https://github.com/openlmlab/collie","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"adan","method_name":"Adan"},{"method_slug":null,"method_name":"Library"},{"method_slug":"lion","method_name":"Lion"},{"method_slug":"sophia","method_name":"Sophia"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.00407","atlas_url":"https://app.syntology.ai/?focus=2312.00407","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.00407"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openlmlab/collie","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/OpenLMLab/collie","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7accb4cc8f6b2131","entry":"inputs_to_kv_cache_for_model","repo":"OpenLMLab/collie","repo_kind":"official","path":"collie/models/utils.py","file_url":"https://github.com/OpenLMLab/collie/blob/HEAD/collie/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7accb4cc8f6b2131"}},{"code_sha256_prefix":"4bdb94ab969119dc","entry":"kv_cache_to_inputs_for_model","repo":"OpenLMLab/collie","repo_kind":"official","path":"collie/models/utils.py","file_url":"https://github.com/OpenLMLab/collie/blob/HEAD/collie/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4bdb94ab969119dc"}},{"code_sha256_prefix":"3240d072cd82bbff","entry":"load_config","repo":"OpenLMLab/collie","repo_kind":"official","path":"collie/config.py","file_url":"https://github.com/OpenLMLab/collie/blob/HEAD/collie/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3240d072cd82bbff"}},{"code_sha256_prefix":"25e3c49687867f6c","entry":"prepare_callback","repo":"OpenLMLab/collie","repo_kind":"official","path":"collie/callbacks/callback_manager.py","file_url":"https://github.com/OpenLMLab/collie/blob/HEAD/collie/callbacks/callback_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25e3c49687867f6c"}},{"code_sha256_prefix":"91010674e84e0e76","entry":"config_command_parser","repo":"OpenLMLab/collie","repo_kind":"official","path":"collie_cli/config.py","file_url":"https://github.com/OpenLMLab/collie/blob/HEAD/collie_cli/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"91010674e84e0e76"}},{"code_sha256_prefix":"f854c14e7ec1871d","entry":"flash_attention","repo":"OpenLMLab/collie","repo_kind":"official","path":"collie/models/utils.py","file_url":"https://github.com/OpenLMLab/collie/blob/HEAD/collie/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f854c14e7ec1871d"}},{"code_sha256_prefix":"605ce2d0520693bc","entry":"generate_deepspeed_config","repo":"OpenLMLab/collie","repo_kind":"official","path":"collie_cli/config.py","file_url":"https://github.com/OpenLMLab/collie/blob/HEAD/collie_cli/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"605ce2d0520693bc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}