{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/node-feature-extraction-by-self-supervised-1","title":"Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction","arxiv_id":"2111.00064","date":"2021-10-29","proceeding":"ICLR 2022 4","authors":["Eli Chien","Wei-Cheng Chang","Cho-Jui Hsieh","Hsiang-Fu Yu","Jiong Zhang","Olgica Milenkovic","Inderjit S Dhillon"],"abstract":"Learning on graphs has attracted significant attention in the learning community due to numerous real-world applications. In particular, graph neural networks (GNNs), which take numerical node features and graph structure as inputs, have been shown to achieve state-of-the-art performance on various graph-related learning tasks. Recent works exploring the correlation between numerical node features and graph structure via self-supervised learning have paved the way for further performance improvements of GNNs. However, methods used for extracting numerical node features from raw data are still graph-agnostic within standard GNN pipelines. This practice is sub-optimal as it prevents one from fully utilizing potential correlations between graph topology and node attributes. To mitigate this issue, we propose a new self-supervised learning framework, Graph Information Aided Node feature exTraction (GIANT). GIANT makes use of the eXtreme Multi-label Classification (XMC) formalism, which is crucial for fine-tuning the language model based on graph information, and scales to large datasets. We also provide a theoretical analysis that justifies the use of XMC over link prediction and motivates integrating XR-Transformers, a powerful method for solving XMC problems, into the GIANT framework. We demonstrate the superior performance of GIANT over the standard GNN pipeline on Open Graph Benchmark datasets: For example, we improve the accuracy of the top-ranked method GAMLP from $68.25\\%$ to $69.67\\%$, SGC from $63.29\\%$ to $66.10\\%$ and MLP from $47.24\\%$ to $61.10\\%$ on the ogbn-papers100M dataset by leveraging GIANT.","url_abs":"https://arxiv.org/abs/2111.00064v3","url_pdf":"https://arxiv.org/pdf/2111.00064v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"node-feature-extraction-by-self-supervised-1","repo_url":"https://github.com/amzn/pecos","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"node-feature-extraction-by-self-supervised-1","repo_url":"https://github.com/OctoberChang/GAMLP?fbclid=IwAR03ugAd7H0U_kQrkihBDMJ-Bxy3siwKVbnBTNkz_IVp5J25L0DFlTK0yfc","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"node-feature-extraction-by-self-supervised-1","repo_url":"https://github.com/elichienxD/SAGN_with_SLE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"node-feature-extraction-by-self-supervised-1","repo_url":"https://github.com/elichienxD/deep_gcns_torch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"extreme-multi-label-classification","task_name":"Extreme Multi-Label Classification"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"link-prediction","task_name":"Link Prediction"},{"task_slug":"multi-label-classification-2","task_name":"MUlTI-LABEL-ClASSIFICATION"},{"task_slug":"multi-label-classification","task_name":"Multi-Label Classification"},{"task_slug":"node-property-prediction","task_name":"Node Property Prediction"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/node-property-prediction-on-ogbn-arxiv","task":"Node Property Prediction","dataset":"ogbn-arxiv","model":"GIANT-XRT+RevGAT+KD (use raw text)","rank_in_archive_order":13,"of":86,"metrics":{"Ext. data":"Yes","Number of params":"1304912","Test Accuracy":"0.7615 ± 0.0010","Validation Accuracy":"0.7716 ± 0.0009"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-arxiv","task":"Node Property Prediction","dataset":"ogbn-arxiv","model":"GIANT-XRT+GraphSAGE (use raw text)","rank_in_archive_order":19,"of":86,"metrics":{"Ext. data":"Yes","Number of params":"546344","Test Accuracy":"0.7435 ± 0.0014","Validation Accuracy":"0.7595 ± 0.0011"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-arxiv","task":"Node Property Prediction","dataset":"ogbn-arxiv","model":"GIANT-XRT+MLP (use raw text)","rank_in_archive_order":50,"of":86,"metrics":{"Ext. data":"Yes","Number of params":"273960","Test Accuracy":"0.7306 ± 0.0011","Validation Accuracy":"0.7432 ± 0.0009"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-papers100m","task":"Node Property Prediction","dataset":"ogbn-papers100M","model":"GIANT-XRT+GAMLP+RLU (use raw text)","rank_in_archive_order":2,"of":20,"metrics":{"Ext. data":"Yes","Number of params":"21551631","Test Accuracy":"0.6967 ± 0.0005","Validation Accuracy":"0.7305 ± 0.0004"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-products","task":"Node Property Prediction","dataset":"ogbn-products","model":"GIANT-XRT+SAGN+SLE+C&S (use raw text)","rank_in_archive_order":12,"of":64,"metrics":{"Ext. data":"Yes","Number of params":"1548382","Test Accuracy":"0.8643 ± 0.0020","Validation Accuracy":"0.9352 ± 0.0005"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-products","task":"Node Property Prediction","dataset":"ogbn-products","model":"GIANT-XRT+SAGN+SLE (use raw text)","rank_in_archive_order":13,"of":64,"metrics":{"Ext. data":"Yes","Number of params":"1548382","Test Accuracy":"0.8622 ± 0.0022","Validation Accuracy":"0.9363 ± 0.0005"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-products","task":"Node Property Prediction","dataset":"ogbn-products","model":"GIANT-XRT+GraphSAINT(use raw text)","rank_in_archive_order":25,"of":64,"metrics":{"Ext. data":"Yes","Number of params":"417583","Test Accuracy":"0.8415 ± 0.0022","Validation Accuracy":"0.9318 ± 0.0004"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-products","task":"Node Property Prediction","dataset":"ogbn-products","model":"GIANT-XRT+MLP (use raw text)","rank_in_archive_order":44,"of":64,"metrics":{"Ext. data":"Yes","Number of params":"275759","Test Accuracy":"0.8049 ± 0.0028","Validation Accuracy":"0.9210 ± 0.0009"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.00064","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.00064"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amzn/pecos","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/elichienxD/SAGN_with_SLE","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/OctoberChang/GAMLP?fbclid=IwAR03ugAd7H0U_kQrkihBDMJ-Bxy3siwKVbnBTNkz_IVp5J25L0DFlTK0yfc","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/elichienxD/deep_gcns_torch","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"ran_draft_wrong":2,"ran_honours":3,"unverified":14},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1},"listed":{"samples":16,"ran":2,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c6acd3e400fc2676","entry":"get_n_params","repo":"elichienxD/SAGN_with_SLE","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/elichienxD/SAGN_with_SLE/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6acd3e400fc2676"}},{"code_sha256_prefix":"09e6ddb17c0adf28","entry":"norm_layer","repo":"elichienxD/deep_gcns_torch","repo_kind":"listed","path":"gcn_lib/dense/torch_nn.py","file_url":"https://github.com/elichienxD/deep_gcns_torch/blob/HEAD/gcn_lib/dense/torch_nn.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09e6ddb17c0adf28"}},{"code_sha256_prefix":"8e6629dff4ea0a71","entry":"test","repo":"amzn/pecos","repo_kind":"official","path":"examples/giant-xrt/OGB_baselines/ogbn-arxiv/gnn.py","file_url":"https://github.com/amzn/pecos/blob/HEAD/examples/giant-xrt/OGB_baselines/ogbn-arxiv/gnn.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8e6629dff4ea0a71"}},{"code_sha256_prefix":"b40f6b945f2312d4","entry":"test","repo":"amzn/pecos","repo_kind":"official","path":"examples/giant-xrt/OGB_baselines/ogbn-papers100M/mlp_xrt.py","file_url":"https://github.com/amzn/pecos/blob/HEAD/examples/giant-xrt/OGB_baselines/ogbn-papers100M/mlp_xrt.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b40f6b945f2312d4"}},{"code_sha256_prefix":"82f54a95d4951b19","entry":"train","repo":"amzn/pecos","repo_kind":"official","path":"examples/giant-xrt/OGB_baselines/ogbn-arxiv/gnn.py","file_url":"https://github.com/amzn/pecos/blob/HEAD/examples/giant-xrt/OGB_baselines/ogbn-arxiv/gnn.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"82f54a95d4951b19"}},{"code_sha256_prefix":"17c5046a153ae409","entry":"train","repo":"amzn/pecos","repo_kind":"official","path":"examples/giant-xrt/OGB_baselines/ogbn-papers100M/mlp_xrt.py","file_url":"https://github.com/amzn/pecos/blob/HEAD/examples/giant-xrt/OGB_baselines/ogbn-papers100M/mlp_xrt.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17c5046a153ae409"}},{"code_sha256_prefix":"81d31aad45664fa2","entry":"act_layer","repo":"elichienxD/deep_gcns_torch","repo_kind":"listed","path":"gcn_lib/dense/torch_nn.py","file_url":"https://github.com/elichienxD/deep_gcns_torch/blob/HEAD/gcn_lib/dense/torch_nn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81d31aad45664fa2"}},{"code_sha256_prefix":"6b65956f32ad8f13","entry":"act_layer","repo":"elichienxD/deep_gcns_torch","repo_kind":"listed","path":"gcn_lib/sparse/torch_nn.py","file_url":"https://github.com/elichienxD/deep_gcns_torch/blob/HEAD/gcn_lib/sparse/torch_nn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b65956f32ad8f13"}},{"code_sha256_prefix":"c52260211d1aa99f","entry":"batched_index_select","repo":"elichienxD/deep_gcns_torch","repo_kind":"listed","path":"gcn_lib/dense/torch_nn.py","file_url":"https://github.com/elichienxD/deep_gcns_torch/blob/HEAD/gcn_lib/dense/torch_nn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c52260211d1aa99f"}},{"code_sha256_prefix":"47e2e99992a9c1aa","entry":"dense_knn_matrix","repo":"elichienxD/deep_gcns_torch","repo_kind":"listed","path":"gcn_lib/dense/torch_edge.py","file_url":"https://github.com/elichienxD/deep_gcns_torch/blob/HEAD/gcn_lib/dense/torch_edge.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47e2e99992a9c1aa"}},{"code_sha256_prefix":"a4d2591506684852","entry":"entropy","repo":"elichienxD/SAGN_with_SLE","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/elichienxD/SAGN_with_SLE/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4d2591506684852"}},{"code_sha256_prefix":"f31615a9e0aa6f97","entry":"generate_preds_path","repo":"elichienxD/SAGN_with_SLE","repo_kind":"listed","path":"src/post_process.py","file_url":"https://github.com/elichienxD/SAGN_with_SLE/blob/HEAD/src/post_process.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f31615a9e0aa6f97"}},{"code_sha256_prefix":"16e18471c8106af0","entry":"knn_graph_matrix","repo":"elichienxD/deep_gcns_torch","repo_kind":"listed","path":"gcn_lib/sparse/torch_edge.py","file_url":"https://github.com/elichienxD/deep_gcns_torch/blob/HEAD/gcn_lib/sparse/torch_edge.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16e18471c8106af0"}},{"code_sha256_prefix":"ed908dafa30d8f12","entry":"knn_matrix","repo":"elichienxD/deep_gcns_torch","repo_kind":"listed","path":"gcn_lib/sparse/torch_edge.py","file_url":"https://github.com/elichienxD/deep_gcns_torch/blob/HEAD/gcn_lib/sparse/torch_edge.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed908dafa30d8f12"}},{"code_sha256_prefix":"eb1a19a333b59f25","entry":"load_output_files","repo":"elichienxD/SAGN_with_SLE","repo_kind":"listed","path":"src/post_process.py","file_url":"https://github.com/elichienxD/SAGN_with_SLE/blob/HEAD/src/post_process.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb1a19a333b59f25"}},{"code_sha256_prefix":"3be91b595771f70b","entry":"load_pretrained_models","repo":"elichienxD/deep_gcns_torch","repo_kind":"listed","path":"utils/ckpt_util.py","file_url":"https://github.com/elichienxD/deep_gcns_torch/blob/HEAD/utils/ckpt_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3be91b595771f70b"}},{"code_sha256_prefix":"c845b58421ba1367","entry":"load_pretrained_optimizer","repo":"elichienxD/deep_gcns_torch","repo_kind":"listed","path":"utils/ckpt_util.py","file_url":"https://github.com/elichienxD/deep_gcns_torch/blob/HEAD/utils/ckpt_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c845b58421ba1367"}},{"code_sha256_prefix":"bc7cc66599be9207","entry":"pairwise_distance","repo":"elichienxD/deep_gcns_torch","repo_kind":"listed","path":"gcn_lib/dense/torch_edge.py","file_url":"https://github.com/elichienxD/deep_gcns_torch/blob/HEAD/gcn_lib/dense/torch_edge.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc7cc66599be9207"}},{"code_sha256_prefix":"8ccc6f05779c9d3f","entry":"test","repo":"elichienxD/SAGN_with_SLE","repo_kind":"listed","path":"src/train_process.py","file_url":"https://github.com/elichienxD/SAGN_with_SLE/blob/HEAD/src/train_process.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ccc6f05779c9d3f"}},{"code_sha256_prefix":"9301ba207b35bab1","entry":"to_scipy","repo":"elichienxD/SAGN_with_SLE","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/elichienxD/SAGN_with_SLE/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9301ba207b35bab1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}