{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/toolqa-a-dataset-for-llm-question-answering-1","title":"ToolQA: A Dataset for LLM Question Answering with External Tools","arxiv_id":"2306.13304","date":"2023-06-23","proceeding":"NeurIPS 2023 11","authors":["Yuchen Zhuang","Yue Yu","Kuan Wang","Haotian Sun","Chao Zhang"],"abstract":"Large Language Models (LLMs) have demonstrated impressive performance in various NLP tasks, but they still suffer from challenges such as hallucination and weak numerical reasoning. To overcome these challenges, external tools can be used to enhance LLMs' question-answering abilities. However, current evaluation methods do not distinguish between questions that can be answered using LLMs' internal knowledge and those that require external information through tool use. To address this issue, we introduce a new dataset called ToolQA, which is designed to faithfully evaluate LLMs' ability to use external tools for question answering. Our development of ToolQA involved a scalable, automated process for dataset curation, along with 13 specialized tools designed for interaction with external knowledge in order to answer questions. Importantly, we strive to minimize the overlap between our benchmark data and LLMs' pre-training data, enabling a more precise evaluation of LLMs' tool-use reasoning abilities. We conducted an in-depth diagnosis of existing tool-use LLMs to highlight their strengths, weaknesses, and potential improvements. Our findings set a new benchmark for evaluating LLMs and suggest new directions for future advancements. Our data and code are freely available to the broader scientific community on GitHub.","url_abs":"https://arxiv.org/abs/2306.13304v1","url_pdf":"https://arxiv.org/pdf/2306.13304v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"toolqa-a-dataset-for-llm-question-answering-1","repo_url":"https://github.com/night-chen/toolqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"toolqa-a-dataset-for-llm-question-answering-1","repo_url":"https://github.com/chen-gx/toolevo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.13304","atlas_url":"https://app.syntology.ai/?focus=2306.13304","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.13304"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/night-chen/toolqa","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chen-gx/toolevo","reach":{"status":"ok"}}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":12},"by_repo_kind":{"official":{"samples":14,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c373c5e580ba5c6f","entry":"clean_str","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/ChatGPT/code/query.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/ChatGPT/code/query.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c373c5e580ba5c6f"}},{"code_sha256_prefix":"84d28fbaf38b0a59","entry":"format_step","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/ReAct/code/agents.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/ReAct/code/agents.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"84d28fbaf38b0a59"}},{"code_sha256_prefix":"898fb37499d60ace","entry":"EM","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/ReAct/code/environment.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/ReAct/code/environment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"898fb37499d60ace"}},{"code_sha256_prefix":"667861b2eb74408c","entry":"dispatch_openai_requests","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/ChatGPT/code/query.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/ChatGPT/code/query.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"667861b2eb74408c"}},{"code_sha256_prefix":"ad24504e2398bb16","entry":"finish","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/chameleon/run_toolqa/model.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/chameleon/run_toolqa/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad24504e2398bb16"}},{"code_sha256_prefix":"08a9a18a15549ce9","entry":"get_codex_response","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/chameleon/utilities.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/chameleon/utilities.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"08a9a18a15549ce9"}},{"code_sha256_prefix":"f7dd05e069f54905","entry":"get_gpt3_response","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/chameleon/utilities.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/chameleon/utilities.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f7dd05e069f54905"}},{"code_sha256_prefix":"3f453625dacae10a","entry":"normalize_answer","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/ReAct/code/environment.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/ReAct/code/environment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3f453625dacae10a"}},{"code_sha256_prefix":"f71b690030237706","entry":"parse_action","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/chameleon/run_toolqa/run-v2.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/chameleon/run_toolqa/run-v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f71b690030237706"}},{"code_sha256_prefix":"0cd0094dec049291","entry":"parse_action","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/ReAct/code/agents.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/ReAct/code/agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0cd0094dec049291"}},{"code_sha256_prefix":"23aab7577a21a4cf","entry":"parse_action","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/ReAct/code/environment.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/ReAct/code/environment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"23aab7577a21a4cf"}},{"code_sha256_prefix":"d228da106025a025","entry":"reactLLMMock","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/ReAct/code/mocks.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/ReAct/code/mocks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d228da106025a025"}},{"code_sha256_prefix":"48168012e82e413b","entry":"reflectLLMMock","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/ReAct/code/mocks.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/ReAct/code/mocks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"48168012e82e413b"}},{"code_sha256_prefix":"d9db4727cc8f0df8","entry":"safe_execute","repo":"night-chen/toolqa","repo_kind":"official","path":"benchmark/chameleon/utilities.py","file_url":"https://github.com/night-chen/toolqa/blob/HEAD/benchmark/chameleon/utilities.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9db4727cc8f0df8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}