{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/knowledge-graph-question-answering-datasets","title":"Knowledge Graph Question Answering Datasets and Their Generalizability: Are They Enough for Future Research?","arxiv_id":"2205.06573","date":"2022-05-13","proceeding":null,"authors":["Longquan Jiang","Ricardo Usbeck"],"abstract":"Existing approaches on Question Answering over Knowledge Graphs (KGQA) have weak generalizability. That is often due to the standard i.i.d. assumption on the underlying dataset. Recently, three levels of generalization for KGQA were defined, namely i.i.d., compositional, zero-shot. We analyze 25 well-known KGQA datasets for 5 different Knowledge Graphs (KGs). We show that according to this definition many existing and online available KGQA datasets are either not suited to train a generalizable KGQA system or that the datasets are based on discontinued and out-dated KGs. Generating new datasets is a costly process and, thus, is not an alternative to smaller research groups and companies. In this work, we propose a mitigation method for re-splitting available KGQA datasets to enable their applicability to evaluate generalization, without any cost and manual effort. We test our hypothesis on three KGQA datasets, i.e., LC-QuAD, LC-QuAD 2.0 and QALD-9). Experiments on re-splitted KGQA datasets demonstrate its effectiveness towards generalizability. The code and a unified way to access 18 available datasets is online at https://github.com/semantic-systems/KGQA-datasets as well as https://github.com/semantic-systems/KGQA-datasets-generalization.","url_abs":"https://arxiv.org/abs/2205.06573v1","url_pdf":"https://arxiv.org/pdf/2205.06573v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"knowledge-graph-question-answering-datasets","repo_url":"https://github.com/KGQA/KGQA-datasets-generalization","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"graph-question-answering","task_name":"Graph Question Answering"},{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"},{"task_slug":"out-of-distribution-generalization","task_name":"Out-of-Distribution Generalization"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2205.06573","atlas_url":"https://app.syntology.ai/?focus=2205.06573","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.06573"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KGQA/KGQA-datasets-generalization","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0afa85180025da7d","entry":"get_functions_from_sparql","repo":"KGQA/KGQA-datasets-generalization","repo_kind":"official","path":"utils/sparql_util.py","file_url":"https://github.com/KGQA/KGQA-datasets-generalization/blob/HEAD/utils/sparql_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0afa85180025da7d"}},{"code_sha256_prefix":"015dcc8b3d6dcfe2","entry":"get_schema_terms","repo":"KGQA/KGQA-datasets-generalization","repo_kind":"official","path":"resplit.py","file_url":"https://github.com/KGQA/KGQA-datasets-generalization/blob/HEAD/resplit.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"015dcc8b3d6dcfe2"}},{"code_sha256_prefix":"8a3692841c81657b","entry":"get_single_terms","repo":"KGQA/KGQA-datasets-generalization","repo_kind":"official","path":"resplit.py","file_url":"https://github.com/KGQA/KGQA-datasets-generalization/blob/HEAD/resplit.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a3692841c81657b"}},{"code_sha256_prefix":"ea09facb8968367c","entry":"get_tasks","repo":"KGQA/KGQA-datasets-generalization","repo_kind":"official","path":"preprocess.py","file_url":"https://github.com/KGQA/KGQA-datasets-generalization/blob/HEAD/preprocess.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ea09facb8968367c"}},{"code_sha256_prefix":"e07a8adef6a30250","entry":"group_schema_terms","repo":"KGQA/KGQA-datasets-generalization","repo_kind":"official","path":"resplit.py","file_url":"https://github.com/KGQA/KGQA-datasets-generalization/blob/HEAD/resplit.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e07a8adef6a30250"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}