{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/liar-liar-logical-mire-a-benchmark-for","title":"Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models","arxiv_id":"2406.12546","date":"2024-06-18","proceeding":null,"authors":["Philipp Mondorf","Barbara Plank"],"abstract":"Knights and knaves problems represent a classic genre of logical puzzles where characters either tell the truth or lie. The objective is to logically deduce each character's identity based on their statements. The challenge arises from the truth-telling or lying behavior, which influences the logical implications of each statement. Solving these puzzles requires not only direct deductions from individual statements, but the ability to assess the truthfulness of statements by reasoning through various hypothetical scenarios. As such, knights and knaves puzzles serve as compelling examples of suppositional reasoning. In this paper, we introduce $\\textit{TruthQuest}$, a benchmark for suppositional reasoning based on the principles of knights and knaves puzzles. Our benchmark presents problems of varying complexity, considering both the number of characters and the types of logical statements involved. Evaluations on $\\textit{TruthQuest}$ show that large language models like Llama 3 and Mixtral-8x7B exhibit significant difficulties solving these tasks. A detailed error analysis of the models' output reveals that lower-performing models exhibit a diverse range of reasoning errors, frequently failing to grasp the concept of truth and lies. In comparison, more proficient models primarily struggle with accurately inferring the logical implications of potentially false statements.","url_abs":"https://arxiv.org/abs/2406.12546v2","url_pdf":"https://arxiv.org/pdf/2406.12546v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"liar-liar-logical-mire-a-benchmark-for","repo_url":"https://github.com/mainlp/TruthQuest","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"CC-BY-SA-4.0"}}],"tasks":[{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"}],"methods":[{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[{"slug":"truthquest","name":"TruthQuest","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.12546","atlas_url":"https://app.syntology.ai/?focus=2406.12546","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12546"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/mainlp/TruthQuest","reach":{"status":"ok","spdx":"CC-BY-SA-4.0"}}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"8ae9aaf535c1971f","entry":"assemble_few_shot_example","repo":"mainlp/TruthQuest","repo_kind":"official","path":"gen_data.py","file_url":"https://github.com/mainlp/TruthQuest/blob/HEAD/gen_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8ae9aaf535c1971f"}},{"code_sha256_prefix":"ac725e68c1bf5a84","entry":"inverse_permut","repo":"mainlp/TruthQuest","repo_kind":"official","path":"metareasoning/dataprep/data_generator.py","file_url":"https://github.com/mainlp/TruthQuest/blob/HEAD/metareasoning/dataprep/data_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ac725e68c1bf5a84"}},{"code_sha256_prefix":"33346b27774b1973","entry":"load_json","repo":"mainlp/TruthQuest","repo_kind":"official","path":"metareasoning/utils/utils.py","file_url":"https://github.com/mainlp/TruthQuest/blob/HEAD/metareasoning/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"33346b27774b1973"}},{"code_sha256_prefix":"a308874da0bd519f","entry":"random_k_sublist","repo":"mainlp/TruthQuest","repo_kind":"official","path":"metareasoning/dataprep/data_generator.py","file_url":"https://github.com/mainlp/TruthQuest/blob/HEAD/metareasoning/dataprep/data_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a308874da0bd519f"}},{"code_sha256_prefix":"1afdf7479ae45b66","entry":"random_permut","repo":"mainlp/TruthQuest","repo_kind":"official","path":"metareasoning/dataprep/data_generator.py","file_url":"https://github.com/mainlp/TruthQuest/blob/HEAD/metareasoning/dataprep/data_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1afdf7479ae45b66"}},{"code_sha256_prefix":"1935e2c60609666f","entry":"reorder_list","repo":"mainlp/TruthQuest","repo_kind":"official","path":"evaluate_conclusion.py","file_url":"https://github.com/mainlp/TruthQuest/blob/HEAD/evaluate_conclusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1935e2c60609666f"}},{"code_sha256_prefix":"c387ce2fd70d9c06","entry":"read_text_file","repo":"mainlp/TruthQuest","repo_kind":"official","path":"metareasoning/utils/utils.py","file_url":"https://github.com/mainlp/TruthQuest/blob/HEAD/metareasoning/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c387ce2fd70d9c06"}},{"code_sha256_prefix":"d05488b912cd3af0","entry":"read_yaml_file","repo":"mainlp/TruthQuest","repo_kind":"official","path":"metareasoning/utils/utils.py","file_url":"https://github.com/mainlp/TruthQuest/blob/HEAD/metareasoning/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d05488b912cd3af0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}