{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-large-language-models-infer-causation","title":"Can Large Language Models Infer Causation from Correlation?","arxiv_id":"2306.05836","date":"2023-06-09","proceeding":null,"authors":["Zhijing Jin","Jiarui Liu","Zhiheng Lyu","Spencer Poff","Mrinmaya Sachan","Rada Mihalcea","Mona Diab","Bernhard Schölkopf"],"abstract":"Causal inference is one of the hallmarks of human intelligence. While the field of CausalNLP has attracted much interest in the recent years, existing causal inference datasets in NLP primarily rely on discovering causality from empirical knowledge (e.g., commonsense knowledge). In this work, we propose the first benchmark dataset to test the pure causal inference skills of large language models (LLMs). Specifically, we formulate a novel task Corr2Cause, which takes a set of correlational statements and determines the causal relationship between the variables. We curate a large-scale dataset of more than 200K samples, on which we evaluate seventeen existing LLMs. Through our experiments, we identify a key shortcoming of LLMs in terms of their causal inference skills, and show that these models achieve almost close to random performance on the task. This shortcoming is somewhat mitigated when we try to re-purpose LLMs for this skill via finetuning, but we find that these models still fail to generalize -- they can only perform causal inference in in-distribution settings when variable names and textual expressions used in the queries are similar to those in the training set, but fail in out-of-distribution settings generated by perturbing these queries. Corr2Cause is a challenging task for LLMs, and would be helpful in guiding future research on improving LLMs' pure reasoning skills and generalizability. Our data is at https://huggingface.co/datasets/causalnlp/corr2cause. Our code is at https://github.com/causalNLP/corr2cause.","url_abs":"https://arxiv.org/abs/2306.05836v3","url_pdf":"https://arxiv.org/pdf/2306.05836v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-large-language-models-infer-causation","repo_url":"https://github.com/causalnlp/corr2cause","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"causal-inference","task_name":"Causal Inference"}],"methods":[{"method_slug":"fail","method_name":"fail"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.05836","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.05836"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/causalnlp/corr2cause","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":8},"by_repo_kind":{"official":{"samples":9,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d9793b16bbddeedb","entry":"generate_prompt","repo":"causalnlp/corr2cause","repo_kind":"official","path":"code/run_alpaca.py","file_url":"https://github.com/causalnlp/corr2cause/blob/HEAD/code/run_alpaca.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9793b16bbddeedb"}},{"code_sha256_prefix":"8a3decee5b54d71d","entry":"RMSELoss","repo":"causalnlp/corr2cause","repo_kind":"official","path":"code/finetune/utils.py","file_url":"https://github.com/causalnlp/corr2cause/blob/HEAD/code/finetune/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a3decee5b54d71d"}},{"code_sha256_prefix":"0125471072372399","entry":"add_args","repo":"causalnlp/corr2cause","repo_kind":"official","path":"code/finetune/utils.py","file_url":"https://github.com/causalnlp/corr2cause/blob/HEAD/code/finetune/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0125471072372399"}},{"code_sha256_prefix":"e44094bd647b0cdb","entry":"get_model","repo":"causalnlp/corr2cause","repo_kind":"official","path":"code/finetune/models.py","file_url":"https://github.com/causalnlp/corr2cause/blob/HEAD/code/finetune/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e44094bd647b0cdb"}},{"code_sha256_prefix":"e1018057f98e58a0","entry":"get_tokenizer","repo":"causalnlp/corr2cause","repo_kind":"official","path":"code/finetune/models.py","file_url":"https://github.com/causalnlp/corr2cause/blob/HEAD/code/finetune/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1018057f98e58a0"}},{"code_sha256_prefix":"ac2dc39bbcffe7e7","entry":"modify_dict_from_dataparallel","repo":"causalnlp/corr2cause","repo_kind":"official","path":"code/finetune/utils.py","file_url":"https://github.com/causalnlp/corr2cause/blob/HEAD/code/finetune/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac2dc39bbcffe7e7"}},{"code_sha256_prefix":"3e06f2ac8a9909be","entry":"num_samples2splits","repo":"causalnlp/corr2cause","repo_kind":"official","path":"code/data_stats.py","file_url":"https://github.com/causalnlp/corr2cause/blob/HEAD/code/data_stats.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e06f2ac8a9909be"}},{"code_sha256_prefix":"a94fe6e02e03f5dc","entry":"read_optimizer","repo":"causalnlp/corr2cause","repo_kind":"official","path":"code/finetune/models.py","file_url":"https://github.com/causalnlp/corr2cause/blob/HEAD/code/finetune/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a94fe6e02e03f5dc"}},{"code_sha256_prefix":"fcc354ce3cc55f43","entry":"subsample","repo":"causalnlp/corr2cause","repo_kind":"official","path":"code/data_postsampling.py","file_url":"https://github.com/causalnlp/corr2cause/blob/HEAD/code/data_postsampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fcc354ce3cc55f43"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}