{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/exploring-graph-tasks-with-pure-llms-a","title":"Exploring Graph Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation","arxiv_id":"2502.18771","date":"2025-02-26","proceeding":null,"authors":["Yuxiang Wang","Xinnan Dai","Wenqi Fan","Yao Ma"],"abstract":"Graph-structured data has become increasingly prevalent across various domains, raising the demand for effective models to handle graph tasks like node classification and link prediction. Traditional graph learning models like Graph Neural Networks (GNNs) have made significant strides, but their capabilities in handling graph data remain limited in certain contexts. In recent years, large language models (LLMs) have emerged as promising candidates for graph tasks, yet most studies focus primarily on performance benchmarks and fail to address their broader potential, including their ability to handle limited data, their transferability across tasks, and their robustness. In this work, we provide a comprehensive exploration of LLMs applied to graph tasks. We evaluate the performance of pure LLMs, including those without parameter optimization and those fine-tuned with instructions, across various scenarios. Our analysis goes beyond accuracy, assessing LLM ability to perform in few-shot/zero-shot settings, transfer across domains, understand graph structures, and demonstrate robustness in challenging scenarios. We conduct extensive experiments with 16 graph learning models alongside 6 LLMs (e.g., Llama3B, GPT-4o, Qwen-plus), comparing their performance on datasets like Cora, PubMed, ArXiv, and Products. Our findings show that LLMs, particularly those with instruction tuning, outperform traditional models in few-shot settings, exhibit strong domain transferability, and demonstrate excellent generalization and robustness. This work offers valuable insights into the capabilities of LLMs for graph learning, highlighting their advantages and potential for real-world applications, and paving the way for future research in this area. Codes and datasets are released in https://github.com/myflashbarry/LLM-benchmarking.","url_abs":"https://arxiv.org/abs/2502.18771v1","url_pdf":"https://arxiv.org/pdf/2502.18771v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"exploring-graph-tasks-with-pure-llms-a","repo_url":"https://github.com/myflashbarry/LLM-benchmarking","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"graph-learning","task_name":"Graph Learning"},{"task_slug":"link-prediction","task_name":"Link Prediction"},{"task_slug":"node-classification","task_name":"Node Classification"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.18771","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.18771"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/myflashbarry/LLM-benchmarking","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fba7d27f4f0b5438","entry":"evaluate_with_neighbor_loader","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"baseline/Arxiv_to_product.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/baseline/Arxiv_to_product.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fba7d27f4f0b5438"}},{"code_sha256_prefix":"ce3a0ae50cac1fb8","entry":"get_completion_from_messages","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"LLM/deepseek_V3/deepseek_v3_for_heterophily_node.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/LLM/deepseek_V3/deepseek_v3_for_heterophily_node.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce3a0ae50cac1fb8"}},{"code_sha256_prefix":"91d87b3d6bcd5bab","entry":"get_completion_from_messages","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"LLM/llama/Llama_Instruct_link.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/LLM/llama/Llama_Instruct_link.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"91d87b3d6bcd5bab"}},{"code_sha256_prefix":"a350b1d6962bd121","entry":"get_completion_from_messages","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"LLM/qwen/qwen_plus_for_link.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/LLM/qwen/qwen_plus_for_link.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a350b1d6962bd121"}},{"code_sha256_prefix":"51b821d0781ed401","entry":"get_completion_from_messages","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"LLM/qwen/qwen_plus_for_node.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/LLM/qwen/qwen_plus_for_node.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51b821d0781ed401"}},{"code_sha256_prefix":"5ee8c70828d72599","entry":"get_data_and_text","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"baseline/Arxiv_to_product.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/baseline/Arxiv_to_product.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ee8c70828d72599"}},{"code_sha256_prefix":"3a0569b001641d5a","entry":"mask_features","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"baseline/Arxiv_to_cora_graphmae.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/baseline/Arxiv_to_cora_graphmae.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3a0569b001641d5a"}},{"code_sha256_prefix":"8f73d2f2f0fad30d","entry":"process_combination","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"LLM/utils.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/LLM/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8f73d2f2f0fad30d"}},{"code_sha256_prefix":"bd51f5433db414b2","entry":"process_combination","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"LLM/llama/Llama_Instruct_node.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/LLM/llama/Llama_Instruct_node.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd51f5433db414b2"}},{"code_sha256_prefix":"4db581253b100901","entry":"random_node_feature_masking","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"baseline/GraphCL_link.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/baseline/GraphCL_link.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4db581253b100901"}},{"code_sha256_prefix":"0feefffe83da0dea","entry":"split_input_text","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"LLM/utils.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/LLM/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0feefffe83da0dea"}},{"code_sha256_prefix":"988ab2719fa85cc9","entry":"split_input_text_link","repo":"myflashbarry/LLM-benchmarking","repo_kind":"official","path":"LLM/utils.py","file_url":"https://github.com/myflashbarry/LLM-benchmarking/blob/HEAD/LLM/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"988ab2719fa85cc9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}