{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cold-causal-reasoning-in-closed-daily","title":"COLD: Causal reasOning in cLosed Daily activities","arxiv_id":"2411.19500","date":"2024-11-29","proceeding":null,"authors":["Abhinav Joshi","Areeb Ahmad","Ashutosh Modi"],"abstract":"Large Language Models (LLMs) have shown state-of-the-art performance in a variety of tasks, including arithmetic and reasoning; however, to gauge the intellectual capabilities of LLMs, causal reasoning has become a reliable proxy for validating a general understanding of the mechanics and intricacies of the world similar to humans. Previous works in natural language processing (NLP) have either focused on open-ended causal reasoning via causal commonsense reasoning (CCR) or framed a symbolic representation-based question answering for theoretically backed-up analysis via a causal inference engine. The former adds an advantage of real-world grounding but lacks theoretically backed-up analysis/validation, whereas the latter is far from real-world grounding. In this work, we bridge this gap by proposing the COLD (Causal reasOning in cLosed Daily activities) framework, which is built upon human understanding of daily real-world activities to reason about the causal nature of events. We show that the proposed framework facilitates the creation of enormous causal queries (~ 9 million) and comes close to the mini-turing test, simulating causal reasoning to evaluate the understanding of a daily real-world task. We evaluate multiple LLMs on the created causal queries and find that causal reasoning is challenging even for activities trivial to humans. We further explore (the causal reasoning abilities of LLMs) using the backdoor criterion to determine the causal strength between events.","url_abs":"https://arxiv.org/abs/2411.19500v1","url_pdf":"https://arxiv.org/pdf/2411.19500v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cold-causal-reasoning-in-closed-daily","repo_url":"https://github.com/Exploration-Lab/COLD","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"causal-inference","task_name":"Causal Inference"},{"task_slug":"commonsense-causal-reasoning","task_name":"Commonsense Causal Reasoning"},{"task_slug":"event-causality-identification","task_name":"Event Causality Identification"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"causal-inference","method_name":"Causal inference"}],"datasets_introduced":[{"slug":"cold-causal-reasoning-in-closed-daily","name":"COLD: Causal Reasoning in Closed Daily Activities","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2411.19500","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.19500"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Exploration-Lab/COLD","reach":null}],"summary":{"ran_draft_wrong":1,"ran":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"e74f77b7ab07422d","entry":"get_question_text","repo":"Exploration-Lab/COLD","repo_kind":"official","path":"prompt-based-evaluation/evaluation_templates/mcqa.py","file_url":"https://github.com/Exploration-Lab/COLD/blob/HEAD/prompt-based-evaluation/evaluation_templates/mcqa.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e74f77b7ab07422d"}},{"code_sha256_prefix":"4ea1daaa6be097ac","entry":"mcqa","repo":"Exploration-Lab/COLD","repo_kind":"official","path":"prompt-based-evaluation/evaluation_templates/mcqa.py","file_url":"https://github.com/Exploration-Lab/COLD/blob/HEAD/prompt-based-evaluation/evaluation_templates/mcqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ea1daaa6be097ac"}},{"code_sha256_prefix":"411f8f58870873b7","entry":"prompt_templates_mcqa","repo":"Exploration-Lab/COLD","repo_kind":"official","path":"prompt-based-evaluation/evaluation_templates/mcqa.py","file_url":"https://github.com/Exploration-Lab/COLD/blob/HEAD/prompt-based-evaluation/evaluation_templates/mcqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"411f8f58870873b7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}