{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/training-graph-neural-networks-with-1000","title":"Training Graph Neural Networks with 1000 Layers","arxiv_id":"2106.07476","date":"2021-06-14","proceeding":null,"authors":["Guohao Li","Matthias Müller","Bernard Ghanem","Vladlen Koltun"],"abstract":"Deep graph neural networks (GNNs) have achieved excellent results on various tasks on increasingly large graph datasets with millions of nodes and edges. However, memory complexity has become a major obstacle when training deep GNNs for practical applications due to the immense number of nodes, edges, and intermediate activations. To improve the scalability of GNNs, prior works propose smart graph sampling or partitioning strategies to train GNNs with a smaller set of nodes or sub-graphs. In this work, we study reversible connections, group convolutions, weight tying, and equilibrium models to advance the memory and parameter efficiency of GNNs. We find that reversible connections in combination with deep network architectures enable the training of overparameterized GNNs that significantly outperform existing methods on multiple datasets. Our models RevGNN-Deep (1001 layers with 80 channels each) and RevGNN-Wide (448 layers with 224 channels each) were both trained on a single commodity GPU and achieve an ROC-AUC of $87.74 \\pm 0.13$ and $88.24 \\pm 0.15$ on the ogbn-proteins dataset. To the best of our knowledge, RevGNN-Deep is the deepest GNN in the literature by one order of magnitude. Please visit our project website https://www.deepgcns.org/arch/gnn1000 for more information.","url_abs":"https://arxiv.org/abs/2106.07476v3","url_pdf":"https://arxiv.org/pdf/2106.07476v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"training-graph-neural-networks-with-1000","repo_url":"https://github.com/lightaime/deep_gcns_torch/tree/master/examples/ogb_eff/ogbn_proteins","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"training-graph-neural-networks-with-1000","repo_url":"https://github.com/lightaime/deep_gcns_torch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"training-graph-neural-networks-with-1000","repo_url":"https://github.com/lightaime/deep_gcns_torch/tree/master/examples/ogb","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"training-graph-neural-networks-with-1000","repo_url":"https://github.com/lightaime/deep_gcns_torch/tree/master/examples/ogb_eff/ogbn_arxiv_dgl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"graph-sampling","task_name":"Graph Sampling"},{"task_slug":"node-property-prediction","task_name":"Node Property Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/node-property-prediction-on-ogbn-arxiv","task":"Node Property Prediction","dataset":"ogbn-arxiv","model":"RevGAT+N.Adj+LabelReuse+SelfKD","rank_in_archive_order":22,"of":86,"metrics":{"Ext. data":"No","Number of params":"2098256","Test Accuracy":"0.7426 ± 0.0017","Validation Accuracy":"0.7497 ± 0.0008"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-arxiv","task":"Node Property Prediction","dataset":"ogbn-arxiv","model":"RevGAT+NormAdj+LabelReuse","rank_in_archive_order":29,"of":86,"metrics":{"Ext. data":"No","Number of params":"2098256","Test Accuracy":"0.7402 ± 0.0018","Validation Accuracy":"0.7501 ± 0.0010"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-products","task":"Node Property Prediction","dataset":"ogbn-products","model":"RevGNN-112","rank_in_archive_order":30,"of":64,"metrics":{"Ext. data":"No","Number of params":"2945007","Test Accuracy":"0.8307 ± 0.0030","Validation Accuracy":"0.9290 ± 0.0007"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-proteins","task":"Node Property Prediction","dataset":"ogbn-proteins","model":"RevGNN-Wide","rank_in_archive_order":4,"of":26,"metrics":{"Ext. data":"No","Number of params":"68471608","Test ROC-AUC":"0.8824 ± 0.0015","Validation ROC-AUC":"0.9450 ± 0.0008"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-proteins","task":"Node Property Prediction","dataset":"ogbn-proteins","model":"RevGNN-Deep","rank_in_archive_order":6,"of":26,"metrics":{"Ext. data":"No","Number of params":"20031384","Test ROC-AUC":"0.8774 ± 0.0013","Validation ROC-AUC":"0.9326 ± 0.0006"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.07476","atlas_url":"https://app.syntology.ai/?focus=2106.07476","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.07476"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lightaime/deep_gcns_torch/tree/master/examples/ogb_eff/ogbn_arxiv_dgl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lightaime/deep_gcns_torch/tree/master/examples/ogb_eff/ogbn_proteins","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lightaime/deep_gcns_torch/tree/master/examples/ogb","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lightaime/deep_gcns_torch","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":10},"by_repo_kind":{"official":{"samples":11,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"09e6ddb17c0adf28","entry":"norm_layer","repo":"lightaime/deep_gcns_torch","repo_kind":"official","path":"gcn_lib/dense/torch_nn.py","file_url":"https://github.com/lightaime/deep_gcns_torch/blob/HEAD/gcn_lib/dense/torch_nn.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09e6ddb17c0adf28"}},{"code_sha256_prefix":"81d31aad45664fa2","entry":"act_layer","repo":"lightaime/deep_gcns_torch","repo_kind":"official","path":"gcn_lib/dense/torch_nn.py","file_url":"https://github.com/lightaime/deep_gcns_torch/blob/HEAD/gcn_lib/dense/torch_nn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81d31aad45664fa2"}},{"code_sha256_prefix":"6b65956f32ad8f13","entry":"act_layer","repo":"lightaime/deep_gcns_torch","repo_kind":"official","path":"gcn_lib/sparse/torch_nn.py","file_url":"https://github.com/lightaime/deep_gcns_torch/blob/HEAD/gcn_lib/sparse/torch_nn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b65956f32ad8f13"}},{"code_sha256_prefix":"c52260211d1aa99f","entry":"batched_index_select","repo":"lightaime/deep_gcns_torch","repo_kind":"official","path":"gcn_lib/dense/torch_nn.py","file_url":"https://github.com/lightaime/deep_gcns_torch/blob/HEAD/gcn_lib/dense/torch_nn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c52260211d1aa99f"}},{"code_sha256_prefix":"7bc52d942ecf4430","entry":"dense_knn_matrix","repo":"lightaime/deep_gcns_torch","repo_kind":"official","path":"gcn_lib/dense/torch_edge.py","file_url":"https://github.com/lightaime/deep_gcns_torch/blob/HEAD/gcn_lib/dense/torch_edge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7bc52d942ecf4430"}},{"code_sha256_prefix":"c69d40b71d37d2d2","entry":"knn_graph_matrix","repo":"lightaime/deep_gcns_torch","repo_kind":"official","path":"gcn_lib/sparse/torch_edge.py","file_url":"https://github.com/lightaime/deep_gcns_torch/blob/HEAD/gcn_lib/sparse/torch_edge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c69d40b71d37d2d2"}},{"code_sha256_prefix":"2f0e8668c137d3aa","entry":"knn_matrix","repo":"lightaime/deep_gcns_torch","repo_kind":"official","path":"gcn_lib/sparse/torch_edge.py","file_url":"https://github.com/lightaime/deep_gcns_torch/blob/HEAD/gcn_lib/sparse/torch_edge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f0e8668c137d3aa"}},{"code_sha256_prefix":"3be91b595771f70b","entry":"load_pretrained_models","repo":"lightaime/deep_gcns_torch","repo_kind":"official","path":"utils/ckpt_util.py","file_url":"https://github.com/lightaime/deep_gcns_torch/blob/HEAD/utils/ckpt_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3be91b595771f70b"}},{"code_sha256_prefix":"c845b58421ba1367","entry":"load_pretrained_optimizer","repo":"lightaime/deep_gcns_torch","repo_kind":"official","path":"utils/ckpt_util.py","file_url":"https://github.com/lightaime/deep_gcns_torch/blob/HEAD/utils/ckpt_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c845b58421ba1367"}},{"code_sha256_prefix":"34c1fc07af3538f8","entry":"norm_layer","repo":"lightaime/deep_gcns_torch","repo_kind":"official","path":"gcn_lib/sparse/torch_nn.py","file_url":"https://github.com/lightaime/deep_gcns_torch/blob/HEAD/gcn_lib/sparse/torch_nn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"34c1fc07af3538f8"}},{"code_sha256_prefix":"bc7cc66599be9207","entry":"pairwise_distance","repo":"lightaime/deep_gcns_torch","repo_kind":"official","path":"gcn_lib/dense/torch_edge.py","file_url":"https://github.com/lightaime/deep_gcns_torch/blob/HEAD/gcn_lib/dense/torch_edge.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc7cc66599be9207"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}