{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/questbench-can-llms-ask-the-right-question-to","title":"QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?","arxiv_id":"2503.22674","date":"2025-03-28","proceeding":null,"authors":["Belinda Z. Li","Been Kim","Zi Wang"],"abstract":"Recently, a large amount of work has focused on improving large language models' (LLMs') performance on reasoning benchmarks such as math and logic. However, past work has largely assumed that tasks are well-defined. In the real world, queries to LLMs are often underspecified, only solvable through acquiring missing information. We formalize this as a constraint satisfaction problem (CSP) with missing variable assignments. Using a special case of this formalism where only one necessary variable assignment is missing, we can rigorously evaluate an LLM's ability to identify the minimal necessary question to ask and quantify axes of difficulty levels for each problem. We present QuestBench, a set of underspecified reasoning tasks solvable by asking at most one question, which includes: (1) Logic-Q: Logical reasoning tasks with one missing proposition, (2) Planning-Q: PDDL planning problems with initial states that are partially-observed, (3) GSM-Q: Human-annotated grade school math problems with one missing variable assignment, and (4) GSME-Q: a version of GSM-Q where word problems are translated into equations by human annotators. The LLM is tasked with selecting the correct clarification question(s) from a list of options. While state-of-the-art models excel at GSM-Q and GSME-Q, their accuracy is only 40-50% on Logic-Q and Planning-Q. Analysis demonstrates that the ability to solve well-specified reasoning problems may not be sufficient for success on our benchmark: models have difficulty identifying the right question to ask, even when they can solve the fully specified version of the problem. Furthermore, in the Planning-Q domain, LLMs tend not to hedge, even when explicitly presented with the option to predict ``not sure.'' This highlights the need for deeper investigation into models' information acquisition capabilities.","url_abs":"https://arxiv.org/abs/2503.22674v1","url_pdf":"https://arxiv.org/pdf/2503.22674v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"questbench-can-llms-ask-the-right-question-to","repo_url":"https://github.com/google-deepmind/questbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"},{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.22674","atlas_url":"https://app.syntology.ai/?focus=2503.22674","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.22674"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/questbench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":11,"unverified":1},"by_repo_kind":{"official":{"samples":12,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5aaad1d50295832c","entry":"check_satisfies_constraints","repo":"google-deepmind/questbench","repo_kind":"official","path":"Planning/backtrace_utils.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/Planning/backtrace_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5aaad1d50295832c"}},{"code_sha256_prefix":"950be6a18b37b85e","entry":"check_self_consistency","repo":"google-deepmind/questbench","repo_kind":"official","path":"Planning/backtrace_utils.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/Planning/backtrace_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"950be6a18b37b85e"}},{"code_sha256_prefix":"938eefb6de7c1539","entry":"jsonify_prompt","repo":"google-deepmind/questbench","repo_kind":"official","path":"model_utils.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"938eefb6de7c1539"}},{"code_sha256_prefix":"08f538194a1bc5fa","entry":"load_cache_file","repo":"google-deepmind/questbench","repo_kind":"official","path":"model_utils.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"08f538194a1bc5fa"}},{"code_sha256_prefix":"dd14497bd4b9e103","entry":"load_data","repo":"google-deepmind/questbench","repo_kind":"official","path":"SimpleLogic/ruleset.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/SimpleLogic/ruleset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd14497bd4b9e103"}},{"code_sha256_prefix":"8d6a2e3dfcad9e26","entry":"make_constraints","repo":"google-deepmind/questbench","repo_kind":"official","path":"Planning/make_heldout_states.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/Planning/make_heldout_states.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d6a2e3dfcad9e26"}},{"code_sha256_prefix":"21b990e2c7f2ae64","entry":"make_heldout_states","repo":"google-deepmind/questbench","repo_kind":"official","path":"Planning/make_heldout_states.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/Planning/make_heldout_states.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"21b990e2c7f2ae64"}},{"code_sha256_prefix":"2b52415e972eca31","entry":"make_impossible_and_contradicting_facts","repo":"google-deepmind/questbench","repo_kind":"official","path":"Planning/make_heldout_states.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/Planning/make_heldout_states.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b52415e972eca31"}},{"code_sha256_prefix":"f1184021116c0ed9","entry":"negate","repo":"google-deepmind/questbench","repo_kind":"official","path":"SimpleLogic/ruleset.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/SimpleLogic/ruleset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f1184021116c0ed9"}},{"code_sha256_prefix":"e661cb08acaf5994","entry":"powerset","repo":"google-deepmind/questbench","repo_kind":"official","path":"Planning/backtrace_utils.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/Planning/backtrace_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e661cb08acaf5994"}},{"code_sha256_prefix":"fe2be5fdd1352cb9","entry":"union","repo":"google-deepmind/questbench","repo_kind":"official","path":"SimpleLogic/derivation.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/SimpleLogic/derivation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe2be5fdd1352cb9"}},{"code_sha256_prefix":"6f3d7c5896fdf158","entry":"openai_request","repo":"google-deepmind/questbench","repo_kind":"official","path":"model_utils.py","file_url":"https://github.com/google-deepmind/questbench/blob/HEAD/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f3d7c5896fdf158"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}