{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/coconet-co-optimizing-computation-and","title":"Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads","arxiv_id":"2105.05720","date":"2021-05-12","proceeding":null,"authors":["Abhinav Jangda","Jun Huang","Guodong Liu","Amir Hossein Nodehi Sabet","Saeed Maleki","Youshan Miao","Madanlal Musuvathi","Todd Mytkowicz","Olli Sarikivi"],"abstract":"Recent trend towards increasing large machine learning models require both training and inference tasks to be distributed. Considering the huge cost of training these models, it is imperative to unlock optimizations in computation and communication to obtain best performance. However, current logical separation between computation and communication kernels in deep learning frameworks misses the optimization opportunities across such barrier. Breaking this abstraction with a holistic consideration can provide many optimizations to provide performance improvements in distributed workloads. Manually applying these optimizations needs modifications in underlying computation and communication libraries for each scenario, which is time consuming and error-prone. Therefore, we present CoCoNeT, with a DSL to express a program with both computation and communication. CoCoNeT contains several machine learning aware transformations to optimize a program and a compiler to generate high performance kernels. Providing both computation and communication as first class constructs allows users to work on a high-level abstraction and apply powerful optimizations, such as fusion or overlapping of communication and computation. CoCoNeT enables us to optimize data-, model-and pipeline-parallel workloads in large language models with only a few lines of code. Experiments show CoCoNeT significantly outperforms state-of-the-art distributed machine learning implementations.","url_abs":"https://arxiv.org/abs/2105.05720v5","url_pdf":"https://arxiv.org/pdf/2105.05720v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"coconet-co-optimizing-computation-and","repo_url":"https://github.com/parasailteam/coconet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"coconet-co-optimizing-computation-and","repo_url":"https://github.com/microsoft/msccl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"}],"methods":[{"method_slug":"aware","method_name":"AWARE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2105.05720","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.05720"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/parasailteam/coconet","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/msccl","reach":null}],"summary":{"ran_draft_wrong":2,"ran_honours":2},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"a5cfd38102d818c4","entry":"execute_command","repo":"parasailteam/coconet","repo_kind":"official","path":"experiments/data-parallel-exp.py","file_url":"https://github.com/parasailteam/coconet/blob/HEAD/experiments/data-parallel-exp.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5cfd38102d818c4"}},{"code_sha256_prefix":"ef704f69e68b5c76","entry":"parse_cpu_clock_scale","repo":"microsoft/msccl","repo_kind":"listed","path":"tools/npkit_trace_generator.py","file_url":"https://github.com/microsoft/msccl/blob/HEAD/tools/npkit_trace_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ef704f69e68b5c76"}},{"code_sha256_prefix":"c2d02803aa225dc6","entry":"parse_gpu_clock_scale","repo":"microsoft/msccl","repo_kind":"listed","path":"tools/npkit_trace_generator.py","file_url":"https://github.com/microsoft/msccl/blob/HEAD/tools/npkit_trace_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c2d02803aa225dc6"}},{"code_sha256_prefix":"47ab52d4f5330e1d","entry":"parse_gpu_event","repo":"microsoft/msccl","repo_kind":"listed","path":"tools/npkit_trace_generator.py","file_url":"https://github.com/microsoft/msccl/blob/HEAD/tools/npkit_trace_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"47ab52d4f5330e1d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}