{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/comparing-inferential-strategies-of-humans","title":"Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning","arxiv_id":"2402.14856","date":"2024-02-20","proceeding":null,"authors":["Philipp Mondorf","Barbara Plank"],"abstract":"Deductive reasoning plays a pivotal role in the formulation of sound and cohesive arguments. It allows individuals to draw conclusions that logically follow, given the truth value of the information provided. Recent progress in the domain of large language models (LLMs) has showcased their capability in executing deductive reasoning tasks. Nonetheless, a significant portion of research primarily assesses the accuracy of LLMs in solving such tasks, often overlooking a deeper analysis of their reasoning behavior. In this study, we draw upon principles from cognitive psychology to examine inferential strategies employed by LLMs, through a detailed evaluation of their responses to propositional logic problems. Our findings indicate that LLMs display reasoning patterns akin to those observed in humans, including strategies like $\\textit{supposition following}$ or $\\textit{chain construction}$. Moreover, our research demonstrates that the architecture and scale of the model significantly affect its preferred method of reasoning, with more advanced models tending to adopt strategies more frequently than less sophisticated ones. Importantly, we assert that a model's accuracy, that is the correctness of its final conclusion, does not necessarily reflect the validity of its reasoning process. This distinction underscores the necessity for more nuanced evaluation procedures in the field.","url_abs":"https://arxiv.org/abs/2402.14856v2","url_pdf":"https://arxiv.org/pdf/2402.14856v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"comparing-inferential-strategies-of-humans","repo_url":"https://github.com/mainlp/inferential-strategies","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"CC-BY-SA-4.0"}}],"tasks":[],"methods":[],"datasets_introduced":[{"slug":"inferential-strategies","name":"Inferential-Strategies","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.14856","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.14856"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/mainlp/inferential-strategies","reach":{"status":"ok","spdx":"CC-BY-SA-4.0"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"c9e9d679ca961f0b","entry":"draw_adjective","repo":"mainlp/inferential-strategies","repo_kind":"official","path":"reasoning_strategies/dataprep/preprocessing.py","file_url":"https://github.com/mainlp/inferential-strategies/blob/HEAD/reasoning_strategies/dataprep/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c9e9d679ca961f0b"}},{"code_sha256_prefix":"a8487406395eacf9","entry":"load_data_from_json","repo":"mainlp/inferential-strategies","repo_kind":"official","path":"reasoning_strategies/utils/utils.py","file_url":"https://github.com/mainlp/inferential-strategies/blob/HEAD/reasoning_strategies/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a8487406395eacf9"}},{"code_sha256_prefix":"3f821e04d5d9a0fe","entry":"read_yaml_file","repo":"mainlp/inferential-strategies","repo_kind":"official","path":"reasoning_strategies/utils/utils.py","file_url":"https://github.com/mainlp/inferential-strategies/blob/HEAD/reasoning_strategies/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3f821e04d5d9a0fe"}},{"code_sha256_prefix":"f496812cd5145189","entry":"replace_words","repo":"mainlp/inferential-strategies","repo_kind":"official","path":"reasoning_strategies/dataprep/preprocessing.py","file_url":"https://github.com/mainlp/inferential-strategies/blob/HEAD/reasoning_strategies/dataprep/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f496812cd5145189"}},{"code_sha256_prefix":"8810006d72cdb240","entry":"starts_with_vowel_sound","repo":"mainlp/inferential-strategies","repo_kind":"official","path":"reasoning_strategies/dataprep/preprocessing.py","file_url":"https://github.com/mainlp/inferential-strategies/blob/HEAD/reasoning_strategies/dataprep/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8810006d72cdb240"}},{"code_sha256_prefix":"c387ce2fd70d9c06","entry":"read_text_file","repo":"mainlp/inferential-strategies","repo_kind":"official","path":"reasoning_strategies/utils/utils.py","file_url":"https://github.com/mainlp/inferential-strategies/blob/HEAD/reasoning_strategies/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c387ce2fd70d9c06"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}