{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-the-combinatorial","title":"Benchmarking the Combinatorial Generalizability of Complex Query Answering on Knowledge Graphs","arxiv_id":"2109.08925","date":"2021-09-18","proceeding":null,"authors":["ZiHao Wang","Hang Yin","Yangqiu Song"],"abstract":"Complex Query Answering (CQA) is an important reasoning task on knowledge graphs. Current CQA learning models have been shown to be able to generalize from atomic operators to more complex formulas, which can be regarded as the combinatorial generalizability. In this paper, we present EFO-1-QA, a new dataset to benchmark the combinatorial generalizability of CQA models by including 301 different queries types, which is 20 times larger than existing datasets. Besides, our work, for the first time, provides a benchmark to evaluate and analyze the impact of different operators and normal forms by using (a) 7 choices of the operator systems and (b) 9 forms of complex queries. Specifically, we provide the detailed study of the combinatorial generalizability of two commonly used operators, i.e., projection and intersection, and justify the impact of the forms of queries given the canonical choice of operators. Our code and data can provide an effective pipeline to benchmark CQA models.","url_abs":"https://arxiv.org/abs/2109.08925v2","url_pdf":"https://arxiv.org/pdf/2109.08925v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-the-combinatorial","repo_url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"benchmarking-the-combinatorial","repo_url":"https://github.com/Jacfger/simple-stuffs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"complex-query-answering","task_name":"Complex Query Answering"},{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"}],"methods":[],"datasets_introduced":[{"slug":"efo-1-qa","name":"EFO-1-QA","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2109.08925","atlas_url":"https://app.syntology.ai/?focus=2109.08925","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.08925"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Jacfger/simple-stuffs","reach":null}],"summary":{"ran_draft_wrong":4,"ran_honours":2,"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fa8f6d2163c007e7","entry":"convert_to_arg","repo":"hkust-knowcomp/efo-1-qa-benchmark","repo_kind":"official","path":"fol/estimator_ConE.py","file_url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark/blob/HEAD/fol/estimator_ConE.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa8f6d2163c007e7"}},{"code_sha256_prefix":"04df086ab269bc1f","entry":"convert_to_axis","repo":"hkust-knowcomp/efo-1-qa-benchmark","repo_kind":"official","path":"fol/estimator_ConE.py","file_url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark/blob/HEAD/fol/estimator_ConE.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04df086ab269bc1f"}},{"code_sha256_prefix":"9862c45a5afd5103","entry":"count_query_depth","repo":"Jacfger/simple-stuffs","repo_kind":"listed","path":"formula_generation.py","file_url":"https://github.com/Jacfger/simple-stuffs/blob/HEAD/formula_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9862c45a5afd5103"}},{"code_sha256_prefix":"d8b0a4e3912306de","entry":"find_optimal_batch","repo":"hkust-knowcomp/efo-1-qa-benchmark","repo_kind":"official","path":"fol/appfoq.py","file_url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark/blob/HEAD/fol/appfoq.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8b0a4e3912306de"}},{"code_sha256_prefix":"9910e2fc297f8665","entry":"identity","repo":"hkust-knowcomp/efo-1-qa-benchmark","repo_kind":"official","path":"fol/estimator_box.py","file_url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark/blob/HEAD/fol/estimator_box.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9910e2fc297f8665"}},{"code_sha256_prefix":"6df8d7def6d8b856","entry":"order_bounds","repo":"hkust-knowcomp/efo-1-qa-benchmark","repo_kind":"official","path":"fol/estimator_logic.py","file_url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark/blob/HEAD/fol/estimator_logic.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6df8d7def6d8b856"}},{"code_sha256_prefix":"51d9d0e0413cd046","entry":"valclamp","repo":"hkust-knowcomp/efo-1-qa-benchmark","repo_kind":"official","path":"fol/estimator_logic.py","file_url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark/blob/HEAD/fol/estimator_logic.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51d9d0e0413cd046"}},{"code_sha256_prefix":"7cbefe713c9e3840","entry":"inclusion_sampling","repo":"hkust-knowcomp/efo-1-qa-benchmark","repo_kind":"official","path":"fol/appfoq.py","file_url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark/blob/HEAD/fol/appfoq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cbefe713c9e3840"}},{"code_sha256_prefix":"c79e3940bf2965d9","entry":"negative_sampling","repo":"hkust-knowcomp/efo-1-qa-benchmark","repo_kind":"official","path":"fol/appfoq.py","file_url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark/blob/HEAD/fol/appfoq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c79e3940bf2965d9"}},{"code_sha256_prefix":"3ee25729d2bff9f9","entry":"recursive_getattr","repo":"hkust-knowcomp/efo-1-qa-benchmark","repo_kind":"official","path":"validation.py","file_url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark/blob/HEAD/validation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ee25729d2bff9f9"}},{"code_sha256_prefix":"56398fd82d294031","entry":"transform_json_query","repo":"hkust-knowcomp/efo-1-qa-benchmark","repo_kind":"official","path":"transform_beta_data.py","file_url":"https://github.com/hkust-knowcomp/efo-1-qa-benchmark/blob/HEAD/transform_beta_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56398fd82d294031"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}