{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ladder-residual-parallelism-aware","title":"Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping","arxiv_id":"2501.06589","date":"2025-01-11","proceeding":null,"authors":["Muru Zhang","Mayank Mishra","Zhongzhu Zhou","William Brandon","Jue Wang","Yoon Kim","Jonathan Ragan-Kelley","Shuaiwen Leon Song","Ben Athiwaratkun","Tri Dao"],"abstract":"Large language model inference is both memory-intensive and time-consuming, often requiring distributed algorithms to efficiently scale. Various model parallelism strategies are used in multi-gpu training and inference to partition computation across multiple devices, reducing memory load and computation time. However, using model parallelism necessitates communication of information between GPUs, which has been a major bottleneck and limits the gains obtained by scaling up the number of devices. We introduce Ladder Residual, a simple architectural modification applicable to all residual-based models that enables straightforward overlapping that effectively hides the latency of communication. Our insight is that in addition to systems optimization, one can also redesign the model architecture to decouple communication from computation. While Ladder Residual can allow communication-computation decoupling in conventional parallelism patterns, we focus on Tensor Parallelism in this paper, which is particularly bottlenecked by its heavy communication. For a Transformer model with 70B parameters, applying Ladder Residual to all its layers can achieve 29% end-to-end wall clock speed up at inference time with TP sharding over 8 devices. We refer the resulting Transformer model as the Ladder Transformer. We train a 1B and 3B Ladder Transformer from scratch and observe comparable performance to a standard dense transformer baseline. We also show that it is possible to convert parts of the Llama-3.1 8B model to our Ladder Residual architecture with minimal accuracy degradation by only retraining for 3B tokens. We release our code for training and inference for easier replication of experiments.","url_abs":"https://arxiv.org/abs/2501.06589v5","url_pdf":"https://arxiv.org/pdf/2501.06589v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ladder-residual-parallelism-aware","repo_url":"https://github.com/mayank31398/ladder-residual-inference","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"speed","method_name":"SPEED"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.06589","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.06589"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mayank31398/ladder-residual-inference","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":3,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"mayank31398/ladder-residual-inference","repo_kind":"official","path":"hf_modeling_utils/modeling_llama_ladder.py","file_url":"https://github.com/mayank31398/ladder-residual-inference/blob/HEAD/hf_modeling_utils/modeling_llama_ladder.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"bac65c3dafaec040","entry":"apply_rotary_pos_emb","repo":"mayank31398/ladder-residual-inference","repo_kind":"official","path":"hf_modeling_utils/modeling_llama_ladder.py","file_url":"https://github.com/mayank31398/ladder-residual-inference/blob/HEAD/hf_modeling_utils/modeling_llama_ladder.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"bac65c3dafaec040"}},{"code_sha256_prefix":"aba86ffbed27ba65","entry":"find_multiple","repo":"mayank31398/ladder-residual-inference","repo_kind":"official","path":"gpt_fast/gpt_ladder_TP.py","file_url":"https://github.com/mayank31398/ladder-residual-inference/blob/HEAD/gpt_fast/gpt_ladder_TP.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"aba86ffbed27ba65"}},{"code_sha256_prefix":"f4d39ec8a89dcfa0","entry":"logits_to_probs","repo":"mayank31398/ladder-residual-inference","repo_kind":"official","path":"benchmark.py","file_url":"https://github.com/mayank31398/ladder-residual-inference/blob/HEAD/benchmark.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f4d39ec8a89dcfa0"}},{"code_sha256_prefix":"5b6f45360aa68edd","entry":"multinomial_sample_one_no_sync","repo":"mayank31398/ladder-residual-inference","repo_kind":"official","path":"benchmark.py","file_url":"https://github.com/mayank31398/ladder-residual-inference/blob/HEAD/benchmark.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5b6f45360aa68edd"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"mayank31398/ladder-residual-inference","repo_kind":"official","path":"hf_modeling_utils/modeling_llama_ladder.py","file_url":"https://github.com/mayank31398/ladder-residual-inference/blob/HEAD/hf_modeling_utils/modeling_llama_ladder.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"e32aa65ed977997d","entry":"apply_rope_scaling","repo":"mayank31398/ladder-residual-inference","repo_kind":"official","path":"gpt_fast/utils.py","file_url":"https://github.com/mayank31398/ladder-residual-inference/blob/HEAD/gpt_fast/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e32aa65ed977997d"}},{"code_sha256_prefix":"1102ffb719fa5cde","entry":"apply_rotary_emb","repo":"mayank31398/ladder-residual-inference","repo_kind":"official","path":"gpt_fast/utils.py","file_url":"https://github.com/mayank31398/ladder-residual-inference/blob/HEAD/gpt_fast/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1102ffb719fa5cde"}},{"code_sha256_prefix":"410ff1b0adf4919f","entry":"precompute_freqs_cis","repo":"mayank31398/ladder-residual-inference","repo_kind":"official","path":"gpt_fast/utils.py","file_url":"https://github.com/mayank31398/ladder-residual-inference/blob/HEAD/gpt_fast/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"410ff1b0adf4919f"}},{"code_sha256_prefix":"d44fdad445a56d5e","entry":"sample","repo":"mayank31398/ladder-residual-inference","repo_kind":"official","path":"benchmark.py","file_url":"https://github.com/mayank31398/ladder-residual-inference/blob/HEAD/benchmark.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d44fdad445a56d5e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}