{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-to-use-graphs-in-rag-a-comprehensive","title":"When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation","arxiv_id":"2506.05690","date":"2025-06-06","proceeding":null,"authors":["Zhishang Xiang","Chuanjie Wu","Qinggang Zhang","Shengyuan Chen","Zijin Hong","Xiao Huang","Jinsong Su"],"abstract":"Graph retrieval-augmented generation (GraphRAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) with external knowledge. It leverages graphs to model the hierarchical structure between specific concepts, enabling more coherent and effective knowledge retrieval for accurate reasoning.Despite its conceptual promise, recent studies report that GraphRAG frequently underperforms vanilla RAG on many real-world tasks. This raises a critical question: Is GraphRAG really effective, and in which scenarios do graph structures provide measurable benefits for RAG systems? To address this, we propose GraphRAG-Bench, a comprehensive benchmark designed to evaluate GraphRAG models onboth hierarchical knowledge retrieval and deep contextual reasoning. GraphRAG-Bench features a comprehensive dataset with tasks of increasing difficulty, coveringfact retrieval, complex reasoning, contextual summarization, and creative generation, and a systematic evaluation across the entire pipeline, from graph constructionand knowledge retrieval to final generation. Leveraging this novel benchmark, we systematically investigate the conditions when GraphRAG surpasses traditional RAG and the underlying reasons for its success, offering guidelines for its practical application. All related resources and analyses are collected for the community at https://github.com/GraphRAG-Bench/GraphRAG-Benchmark.","url_abs":"https://arxiv.org/abs/2506.05690v1","url_pdf":"https://arxiv.org/pdf/2506.05690v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-to-use-graphs-in-rag-a-comprehensive","repo_url":"https://github.com/graphrag-bench/graphrag-benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.05690","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.05690"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/graphrag-bench/graphrag-benchmark","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"951c176b94644748","entry":"analyze_graph","repo":"graphrag-bench/graphrag-benchmark","repo_kind":"official","path":"Evaluation/indexing_eval.py","file_url":"https://github.com/graphrag-bench/graphrag-benchmark/blob/HEAD/Evaluation/indexing_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"951c176b94644748"}},{"code_sha256_prefix":"f677b555ad8ab5b1","entry":"compute_rouge_score","repo":"graphrag-bench/graphrag-benchmark","repo_kind":"official","path":"Evaluation/metrics/rouge.py","file_url":"https://github.com/graphrag-bench/graphrag-benchmark/blob/HEAD/Evaluation/metrics/rouge.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f677b555ad8ab5b1"}},{"code_sha256_prefix":"213429fc11fc91e3","entry":"fbeta_score","repo":"graphrag-bench/graphrag-benchmark","repo_kind":"official","path":"Evaluation/metrics/answer_accuracy.py","file_url":"https://github.com/graphrag-bench/graphrag-benchmark/blob/HEAD/Evaluation/metrics/answer_accuracy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"213429fc11fc91e3"}},{"code_sha256_prefix":"b24708b5973868ba","entry":"load_graph_from_parquet","repo":"graphrag-bench/graphrag-benchmark","repo_kind":"official","path":"Evaluation/indexing_eval.py","file_url":"https://github.com/graphrag-bench/graphrag-benchmark/blob/HEAD/Evaluation/indexing_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b24708b5973868ba"}},{"code_sha256_prefix":"bb4b5b401765984f","entry":"load_graph_from_pickle","repo":"graphrag-bench/graphrag-benchmark","repo_kind":"official","path":"Evaluation/indexing_eval.py","file_url":"https://github.com/graphrag-bench/graphrag-benchmark/blob/HEAD/Evaluation/indexing_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb4b5b401765984f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}