{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/chatkbqa-a-generate-then-retrieve-framework","title":"ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models","arxiv_id":"2310.08975","date":"2023-10-13","proceeding":null,"authors":["Haoran Luo","Haihong E","Zichen Tang","Shiyao Peng","Yikai Guo","Wentai Zhang","Chenghao Ma","Guanting Dong","Meina Song","Wei Lin","Yifan Zhu","Luu Anh Tuan"],"abstract":"Knowledge Base Question Answering (KBQA) aims to answer natural language questions over large-scale knowledge bases (KBs), which can be summarized into two crucial steps: knowledge retrieval and semantic parsing. However, three core challenges remain: inefficient knowledge retrieval, mistakes of retrieval adversely impacting semantic parsing, and the complexity of previous KBQA methods. To tackle these challenges, we introduce ChatKBQA, a novel and simple generate-then-retrieve KBQA framework, which proposes first generating the logical form with fine-tuned LLMs, then retrieving and replacing entities and relations with an unsupervised retrieval method, to improve both generation and retrieval more directly. Experimental results show that ChatKBQA achieves new state-of-the-art performance on standard KBQA datasets, WebQSP, and CWQ. This work can also be regarded as a new paradigm for combining LLMs with knowledge graphs (KGs) for interpretable and knowledge-required question answering. Our code is publicly available.","url_abs":"https://arxiv.org/abs/2310.08975v3","url_pdf":"https://arxiv.org/pdf/2310.08975v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"chatkbqa-a-generate-then-retrieve-framework","repo_url":"https://github.com/lhrlab/chatkbqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"knowledge-base-question-answering","task_name":"Knowledge Base Question Answering"},{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"semantic-parsing","task_name":"Semantic Parsing"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/knowledge-base-question-answering-on","task":"Knowledge Base Question Answering","dataset":"ComplexWebQuestions","model":"ChatKBQA","rank_in_archive_order":1,"of":6,"metrics":{"Accuracy":"76.8","F1":"81.3","Hits@1":"86.0"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-base-question-answering-on-1","task":"Knowledge Base Question Answering","dataset":"WebQuestionsSP","model":"ChatKBQA","rank_in_archive_order":1,"of":8,"metrics":{"Accuracy":"77.8","F1":"83.5","Hits@1":"86.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.08975","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.08975"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lhrlab/chatkbqa","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran":4,"ran_violates":1,"unverified":8},"by_repo_kind":{"official":{"samples":16,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9df40357afea56cc","entry":"to_list","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"config.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/config.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9df40357afea56cc"}},{"code_sha256_prefix":"7c96e3db8ff7299a","entry":"date_post_process","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"eval_final.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/eval_final.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c96e3db8ff7299a"}},{"code_sha256_prefix":"e73dc55f0b4f1aed","entry":"extract_entities","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"components/expr_parser.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/components/expr_parser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e73dc55f0b4f1aed"}},{"code_sha256_prefix":"eb0198ee05eaa775","entry":"extract_relations","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"components/expr_parser.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/components/expr_parser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb0198ee05eaa775"}},{"code_sha256_prefix":"b2feca00ae40a160","entry":"is_number","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"eval_final.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/eval_final.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2feca00ae40a160"}},{"code_sha256_prefix":"7600856fdde258c1","entry":"prepare_dataloader","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"run_generator_final.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/run_generator_final.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7600856fdde258c1"}},{"code_sha256_prefix":"75b1fc99101be3e8","entry":"tokenize_s_expr","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"components/expr_parser.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/components/expr_parser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75b1fc99101be3e8"}},{"code_sha256_prefix":"038bd5a70277091b","entry":"type_checker","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"eval_final.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/eval_final.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"038bd5a70277091b"}},{"code_sha256_prefix":"6f0de9ba1399e587","entry":"convert_cwq_sparql_instance","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"parse_sparql_cwq.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/parse_sparql_cwq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f0de9ba1399e587"}},{"code_sha256_prefix":"b6fede6aafdf9760","entry":"convert_parse_instance","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"parse_sparql_webqsp.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/parse_sparql_webqsp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6fede6aafdf9760"}},{"code_sha256_prefix":"0632af12ebb615dc","entry":"convert_webqsp_sparql_instance","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"parse_sparql_webqsp.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/parse_sparql_webqsp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0632af12ebb615dc"}},{"code_sha256_prefix":"e267b92fe2e9c426","entry":"get_model_class","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"config.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/config.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e267b92fe2e9c426"}},{"code_sha256_prefix":"2bc45468ebfe2d61","entry":"httpPort","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"Freebase-Setup/virtuoso.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/Freebase-Setup/virtuoso.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2bc45468ebfe2d61"}},{"code_sha256_prefix":"f42bcb1c173e1a09","entry":"isqlPort","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"Freebase-Setup/virtuoso.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/Freebase-Setup/virtuoso.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f42bcb1c173e1a09"}},{"code_sha256_prefix":"e8cde2d16a8c1b7c","entry":"load_untrained_model","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"config.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/config.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8cde2d16a8c1b7c"}},{"code_sha256_prefix":"acd5c28102430f1b","entry":"run","repo":"lhrlab/chatkbqa","repo_kind":"official","path":"Freebase-Setup/virtuoso.py","file_url":"https://github.com/lhrlab/chatkbqa/blob/HEAD/Freebase-Setup/virtuoso.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acd5c28102430f1b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}