{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/redco-a-lightweight-tool-to-automate","title":"RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs","arxiv_id":"2310.16355","date":"2023-10-25","proceeding":null,"authors":["Bowen Tan","Yun Zhu","Lijuan Liu","Hongyi Wang","Yonghao Zhuang","Jindong Chen","Eric Xing","Zhiting Hu"],"abstract":"The recent progress of AI can be largely attributed to large language models (LLMs). However, their escalating memory requirements introduce challenges for machine learning (ML) researchers and engineers. Addressing this requires developers to partition a large model to distribute it across multiple GPUs or TPUs. This necessitates considerable coding and intricate configuration efforts with existing model parallel tools, such as Megatron-LM, DeepSpeed, and Alpa. These tools require users' expertise in machine learning systems (MLSys), creating a bottleneck in LLM development, particularly for developers without MLSys background. In this work, we present RedCoast (Redco), a lightweight and user-friendly tool crafted to automate distributed training and inference for LLMs, as well as to simplify ML pipeline development. The design of Redco emphasizes two key aspects. Firstly, to automate model parallelism, our study identifies two straightforward rules to generate tensor parallel strategies for any given LLM. Integrating these rules into Redco facilitates effortless distributed LLM training and inference, eliminating the need of additional coding or complex configurations. We demonstrate the effectiveness by applying Redco on a set of LLM architectures, such as GPT-J, LLaMA, T5, and OPT, up to the size of 66B. Secondly, we propose a mechanism that allows for the customization of diverse ML pipelines through the definition of merely three functions, avoiding redundant and formulaic code like multi-host related processing. This mechanism proves adaptable across a spectrum of ML algorithms, from foundational language modeling to complex algorithms like meta-learning and reinforcement learning. As a result, Redco implementations exhibit significantly fewer lines of code compared to their official counterparts.","url_abs":"https://arxiv.org/abs/2310.16355v3","url_pdf":"https://arxiv.org/pdf/2310.16355v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"redco-a-lightweight-tool-to-automate","repo_url":"https://github.com/tanyuqian/redco","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"meta-learning","task_name":"Meta-Learning"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"glu","method_name":"Gated Linear Unit"},{"method_slug":"inverse-square-root-schedule","method_name":"Inverse Square Root Schedule"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"opt","method_name":"OPT"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"t5","method_name":"T5"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.16355","atlas_url":"https://app.syntology.ai/?focus=2310.16355","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.16355"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tanyuqian/redco","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":12,"unverified":5},"by_repo_kind":{"official":{"samples":17,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7f7471520b9c1e90","entry":"add_idxes","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/predictors/utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/predictors/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f7471520b9c1e90"}},{"code_sha256_prefix":"ea2c12ce7fd1ec53","entry":"collate_fn_wrapper","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/predictors/utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/predictors/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ea2c12ce7fd1ec53"}},{"code_sha256_prefix":"12b3e76802e87387","entry":"get_data_batches","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/data_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12b3e76802e87387"}},{"code_sha256_prefix":"d0f722215b22ccd3","entry":"get_dataloader","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/data_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d0f722215b22ccd3"}},{"code_sha256_prefix":"10a72bd31f42a074","entry":"get_dtype","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/ckpt_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/ckpt_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10a72bd31f42a074"}},{"code_sha256_prefix":"957b81e4437fbde6","entry":"get_host_examples","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/data_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"957b81e4437fbde6"}},{"code_sha256_prefix":"1879d59e1c180941","entry":"get_logger","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/log_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/log_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1879d59e1c180941"}},{"code_sha256_prefix":"2688afb7a47925ea","entry":"get_lr_schedule_fn","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/opt_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/opt_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2688afb7a47925ea"}},{"code_sha256_prefix":"2e474ceda07a5811","entry":"load_params_shape","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/ckpt_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/ckpt_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e474ceda07a5811"}},{"code_sha256_prefix":"546098c0795323a5","entry":"pred_fn_wrapper","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/predictors/utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/predictors/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"546098c0795323a5"}},{"code_sha256_prefix":"f66310247f3da124","entry":"save_outputs","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/log_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/log_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f66310247f3da124"}},{"code_sha256_prefix":"0e68538ba4cd72e2","entry":"set_partitions","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/partition_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/partition_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e68538ba4cd72e2"}},{"code_sha256_prefix":"89d887e9e94260f0","entry":"default_train_step","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/trainers/utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/trainers/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"89d887e9e94260f0"}},{"code_sha256_prefix":"4925175b59c41e2d","entry":"eval_step","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/trainers/utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/trainers/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4925175b59c41e2d"}},{"code_sha256_prefix":"9d2ee7610dbfcffd","entry":"get_mesh","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/partition_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/partition_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d2ee7610dbfcffd"}},{"code_sha256_prefix":"90b07cf07631b623","entry":"get_params_spec","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/partition_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/partition_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"90b07cf07631b623"}},{"code_sha256_prefix":"dd029e53ad532a38","entry":"load_ckpt","repo":"tanyuqian/redco","repo_kind":"official","path":"redco/deployers/ckpt_utils.py","file_url":"https://github.com/tanyuqian/redco/blob/HEAD/redco/deployers/ckpt_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd029e53ad532a38"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}