{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unveiling-causal-reasoning-in-large-language","title":"Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?","arxiv_id":"2506.21215","date":"2025-06-26","proceeding":null,"authors":["Haoang Chi","He Li","Wenjing Yang","Feng Liu","Long Lan","Xiaoguang Ren","Tongliang Liu","Bo Han"],"abstract":"Causal reasoning capability is critical in advancing large language models (LLMs) toward strong artificial intelligence. While versatile LLMs appear to have demonstrated capabilities in understanding contextual causality and providing responses that obey the laws of causality, it remains unclear whether they perform genuine causal reasoning akin to humans. However, current evidence indicates the contrary. Specifically, LLMs are only capable of performing shallow (level-1) causal reasoning, primarily attributed to the causal knowledge embedded in their parameters, but they lack the capacity for genuine human-like (level-2) causal reasoning. To support this hypothesis, methodologically, we delve into the autoregression mechanism of transformer-based LLMs, revealing that it is not inherently causal. Empirically, we introduce a new causal Q&A benchmark called CausalProbe-2024, whose corpora are fresh and nearly unseen for the studied LLMs. The LLMs exhibit a significant performance drop on CausalProbe-2024 compared to earlier benchmarks, indicating the fact that they primarily engage in level-1 causal reasoning. To bridge the gap towards level-2 causal reasoning, we draw inspiration from the fact that human reasoning is usually facilitated by general knowledge and intended goals. We propose G^2-Reasoner, a method that incorporates general knowledge and goal-oriented prompts into LLMs' causal reasoning processes. Experiments demonstrate that G^2-Reasoner significantly enhances LLMs' causal reasoning capability, particularly in fresh and counterfactual contexts. This work sheds light on a new path for LLMs to advance towards genuine causal reasoning, going beyond level-1 and making strides towards level-2.","url_abs":"https://arxiv.org/abs/2506.21215v1","url_pdf":"https://arxiv.org/pdf/2506.21215v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unveiling-causal-reasoning-in-large-language","repo_url":"https://github.com/haoang97/causalprobe-2024","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"general-knowledge","task_name":"General Knowledge"},{"task_slug":null,"task_name":"counterfactual"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.21215","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.21215"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Haoang97/CausalProbe-2024","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haoang97/causalprobe-2024","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran_draft_wrong":2,"unverified":9},"by_repo_kind":{"official":{"samples":12,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"01971cc37e6fe583","entry":"gather_nograd","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"src/dist_utils.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/src/dist_utils.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01971cc37e6fe583"}},{"code_sha256_prefix":"0c3de3890cd0b382","entry":"load_data","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"passage_retrieval.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/passage_retrieval.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c3de3890cd0b382"}},{"code_sha256_prefix":"6829ceb72569fa12","entry":"metric_max_over_ground_truths","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/metrics.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6829ceb72569fa12"}},{"code_sha256_prefix":"e76cf27d36723be3","entry":"accuracy","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e76cf27d36723be3"}},{"code_sha256_prefix":"7a4988952dd3a127","entry":"calculate_matches","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"src/evaluation.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/src/evaluation.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a4988952dd3a127"}},{"code_sha256_prefix":"637b51b8df78832a","entry":"check_answer","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"src/evaluation.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/src/evaluation.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"637b51b8df78832a"}},{"code_sha256_prefix":"d97ab73d08dfa3b9","entry":"exact_match_score","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d97ab73d08dfa3b9"}},{"code_sha256_prefix":"9fb6e57668661c38","entry":"gather","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"src/dist_utils.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/src/dist_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fb6e57668661c38"}},{"code_sha256_prefix":"fa4ecbff4b9a354b","entry":"has_answer","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"src/evaluation.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/src/evaluation.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa4ecbff4b9a354b"}},{"code_sha256_prefix":"806114c1fae3d57e","entry":"normalize","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"src/normalize_text.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/src/normalize_text.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"806114c1fae3d57e"}},{"code_sha256_prefix":"e037474365501512","entry":"randomcrop","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/src/data.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e037474365501512"}},{"code_sha256_prefix":"b1ad8cd3a7168b7f","entry":"varsize_gather_nograd","repo":"Haoang97/CausalProbe-2024","repo_kind":"official","path":"src/dist_utils.py","file_url":"https://github.com/Haoang97/CausalProbe-2024/blob/HEAD/src/dist_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1ad8cd3a7168b7f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}