{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/crab-assessing-the-strength-of-causal","title":"CRAB: Assessing the Strength of Causal Relationships Between Real-world Events","arxiv_id":"2311.04284","date":"2023-11-07","proceeding":null,"authors":["Angelika Romanou","Syrielle Montariol","Debjit Paul","Leo Laugier","Karl Aberer","Antoine Bosselut"],"abstract":"Understanding narratives requires reasoning about the cause-and-effect relationships between events mentioned in the text. While existing foundation models yield impressive results in many NLP tasks requiring reasoning, it is unclear whether they understand the complexity of the underlying network of causal relationships of events in narratives. In this work, we present CRAB, a new Causal Reasoning Assessment Benchmark designed to evaluate causal understanding of events in real-world narratives. CRAB contains fine-grained, contextual causality annotations for ~2.7K pairs of real-world events that describe various newsworthy event timelines (e.g., the acquisition of Twitter by Elon Musk). Using CRAB, we measure the performance of several large language models, demonstrating that most systems achieve poor performance on the task. Motivated by classical causal principles, we also analyze the causal structures of groups of events in CRAB, and find that models perform worse on causal reasoning when events are derived from complex causal structures compared to simple linear causal chains. We make our dataset and code available to the research community.","url_abs":"https://arxiv.org/abs/2311.04284v1","url_pdf":"https://arxiv.org/pdf/2311.04284v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"crab-assessing-the-strength-of-causal","repo_url":"https://github.com/agromanou/crab","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"crab-assessing-the-strength-of-causal","repo_url":"https://github.com/epfl-nlp/crab","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2311.04284","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.04284"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/agromanou/crab","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/epfl-nlp/crab","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/epfl-nlp/CRAB","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"c6146ed2a3f46646","entry":"ask_gpt3","repo":"epfl-nlp/CRAB","repo_kind":"listed","path":"src/pairwise_causality.py","file_url":"https://github.com/epfl-nlp/CRAB/blob/HEAD/src/pairwise_causality.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6146ed2a3f46646"}},{"code_sha256_prefix":"ce2277196a180d99","entry":"ask_gpt_3","repo":"epfl-nlp/CRAB","repo_kind":"listed","path":"src/graded_causality.py","file_url":"https://github.com/epfl-nlp/CRAB/blob/HEAD/src/graded_causality.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce2277196a180d99"}},{"code_sha256_prefix":"c86a61bd1a02253f","entry":"ask_gpt_4","repo":"epfl-nlp/CRAB","repo_kind":"listed","path":"src/graded_causality.py","file_url":"https://github.com/epfl-nlp/CRAB/blob/HEAD/src/graded_causality.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c86a61bd1a02253f"}},{"code_sha256_prefix":"f8c65c92ec213846","entry":"read_json","repo":"agromanou/crab","repo_kind":"official","path":"src/graded_causality.py","file_url":"https://github.com/agromanou/crab/blob/HEAD/src/graded_causality.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8c65c92ec213846"}},{"code_sha256_prefix":"1eb5fe8a701c4c14","entry":"ask_gpt_4","repo":"epfl-nlp/CRAB","repo_kind":"listed","path":"src/pairwise_causality.py","file_url":"https://github.com/epfl-nlp/CRAB/blob/HEAD/src/pairwise_causality.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1eb5fe8a701c4c14"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}