{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reducing-communication-in-graph-neural","title":"Reducing Communication in Graph Neural Network Training","arxiv_id":"2005.03300","date":"2020-05-07","proceeding":null,"authors":["Alok Tripathy","Katherine Yelick","Aydin Buluc"],"abstract":"Graph Neural Networks (GNNs) are powerful and flexible neural networks that use the naturally sparse connectivity information of the data. GNNs represent this connectivity as sparse matrices, which have lower arithmetic intensity and thus higher communication costs compared to dense matrices, making GNNs harder to scale to high concurrencies than convolutional or fully-connected neural networks. We introduce a family of parallel algorithms for training GNNs and show that they can asymptotically reduce communication compared to previous parallel GNN training methods. We implement these algorithms, which are based on 1D, 1.5D, 2D, and 3D sparse-dense matrix multiplication, using torch.distributed on GPU-equipped clusters. Our algorithms optimize communication across the full GNN training pipeline. We train GNNs on over a hundred GPUs on multiple datasets, including a protein network with over a billion edges.","url_abs":"https://arxiv.org/abs/2005.03300v3","url_pdf":"https://arxiv.org/pdf/2005.03300v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reducing-communication-in-graph-neural","repo_url":"https://github.com/PASSIONLab/CAGNET","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"reducing-communication-in-graph-neural","repo_url":"https://github.com/PASSIONLab/gnn_training","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"graph-neural-network","task_name":"Graph Neural Network"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2005.03300","atlas_url":"https://app.syntology.ai/?focus=2005.03300","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.03300"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PASSIONLab/CAGNET","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PASSIONLab/gnn_training","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"100bf3f3537a255a","entry":"normalize","repo":"PASSIONLab/CAGNET","repo_kind":"official","path":"gcn_distr.py","file_url":"https://github.com/PASSIONLab/CAGNET/blob/HEAD/gcn_distr.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"100bf3f3537a255a"}},{"code_sha256_prefix":"dccec345876296ab","entry":"stop_time","repo":"PASSIONLab/CAGNET","repo_kind":"official","path":"gcn_distr.py","file_url":"https://github.com/PASSIONLab/CAGNET/blob/HEAD/gcn_distr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dccec345876296ab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}