{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generate-on-graph-treat-llm-as-both-agent-and","title":"Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering","arxiv_id":"2404.14741","date":"2024-04-23","proceeding":null,"authors":["Yao Xu","Shizhu He","Jiabei Chen","ZiHao Wang","Yangqiu Song","Hanghang Tong","Guang Liu","Kang Liu","Jun Zhao"],"abstract":"To address the issues of insufficient knowledge and hallucination in Large Language Models (LLMs), numerous studies have explored integrating LLMs with Knowledge Graphs (KGs). However, these methods are typically evaluated on conventional Knowledge Graph Question Answering (KGQA) with complete KGs, where all factual triples required for each question are entirely covered by the given KG. In such cases, LLMs primarily act as an agent to find answer entities within the KG, rather than effectively integrating the internal knowledge of LLMs and external knowledge sources such as KGs. In fact, KGs are often incomplete to cover all the knowledge required to answer questions. To simulate these real-world scenarios and evaluate the ability of LLMs to integrate internal and external knowledge, we propose leveraging LLMs for QA under Incomplete Knowledge Graph (IKGQA), where the provided KG lacks some of the factual triples for each question, and construct corresponding datasets. To handle IKGQA, we propose a training-free method called Generate-on-Graph (GoG), which can generate new factual triples while exploring KGs. Specifically, GoG performs reasoning through a Thinking-Searching-Generating framework, which treats LLM as both Agent and KG in IKGQA. Experimental results on two datasets demonstrate that our GoG outperforms all previous methods.","url_abs":"https://arxiv.org/abs/2404.14741v3","url_pdf":"https://arxiv.org/pdf/2404.14741v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generate-on-graph-treat-llm-as-both-agent-and","repo_url":"https://github.com/yaooxu/gog","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"graph-question-answering","task_name":"Graph Question Answering"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.14741","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.14741"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yaooxu/gog","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/YaooXu/GoG","reach":{"status":"ok"}}],"summary":{"ran":9,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"7eb613aaa936ca12","entry":"add_ns","repo":"YaooXu/GoG","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/YaooXu/GoG/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7eb613aaa936ca12"}},{"code_sha256_prefix":"082ddbc97a2c1ef3","entry":"chunkify_file","repo":"YaooXu/GoG","repo_kind":"official","path":"src/bm25_name2ids.py","file_url":"https://github.com/YaooXu/GoG/blob/HEAD/src/bm25_name2ids.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"082ddbc97a2c1ef3"}},{"code_sha256_prefix":"2381af664b98ca81","entry":"convert_path_to_edge_list","repo":"YaooXu/GoG","repo_kind":"official","path":"src/generate_samples_with_crucial_edges_by_prob.py","file_url":"https://github.com/YaooXu/GoG/blob/HEAD/src/generate_samples_with_crucial_edges_by_prob.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2381af664b98ca81"}},{"code_sha256_prefix":"3c5d4b2cf9c3a681","entry":"convert_path_to_edge_set","repo":"YaooXu/GoG","repo_kind":"official","path":"src/generate_samples_with_crucial_edges_by_prob.py","file_url":"https://github.com/YaooXu/GoG/blob/HEAD/src/generate_samples_with_crucial_edges_by_prob.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c5d4b2cf9c3a681"}},{"code_sha256_prefix":"d37ef44866e11f1d","entry":"exact_match","repo":"YaooXu/GoG","repo_kind":"official","path":"src/evaluate.py","file_url":"https://github.com/YaooXu/GoG/blob/HEAD/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d37ef44866e11f1d"}},{"code_sha256_prefix":"30075cb1dd86dfc3","entry":"process_chunk","repo":"YaooXu/GoG","repo_kind":"official","path":"src/bm25_name2ids.py","file_url":"https://github.com/YaooXu/GoG/blob/HEAD/src/bm25_name2ids.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30075cb1dd86dfc3"}},{"code_sha256_prefix":"8fe8a9776f3ea53c","entry":"remove_ns","repo":"YaooXu/GoG","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/YaooXu/GoG/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8fe8a9776f3ea53c"}},{"code_sha256_prefix":"2ea9bae86f080d5a","entry":"remove_ns_prefix","repo":"YaooXu/GoG","repo_kind":"official","path":"src/generate_samples_with_crucial_edges_by_prob.py","file_url":"https://github.com/YaooXu/GoG/blob/HEAD/src/generate_samples_with_crucial_edges_by_prob.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ea9bae86f080d5a"}},{"code_sha256_prefix":"efa42a65f4c8ffad","entry":"set_environment_variable","repo":"YaooXu/GoG","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/YaooXu/GoG/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efa42a65f4c8ffad"}},{"code_sha256_prefix":"00b2e6d240c0d36b","entry":"prepare_dataset","repo":"YaooXu/GoG","repo_kind":"official","path":"CoT/utils.py","file_url":"https://github.com/YaooXu/GoG/blob/HEAD/CoT/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00b2e6d240c0d36b"}},{"code_sha256_prefix":"b0aab5b997a57a62","entry":"run_llm","repo":"YaooXu/GoG","repo_kind":"official","path":"CoT/utils.py","file_url":"https://github.com/YaooXu/GoG/blob/HEAD/CoT/utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0aab5b997a57a62"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}