{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-08093","title":"CausalDS: Benchmarking Causal Reasoning in Data-Science Agents","arxiv_id":"2607.08093","date":"2026-07-09","proceeding":null,"authors":["Andrej Leban","Yuekai Sun"],"abstract":"Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce CausalDS, a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a scene consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a realistic domain. We optionally ground the composition of the benchmark components in empirical distributions obtained from real-world datasets, thus retaining empirical structure while reducing the \"causal parrot\" risk through completely synthetic generation. From each scene, we then derive tasks spanning all three of Pearl's rungs, with typical data-science prediction tasks appearing as Rung 1. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to the frequent presence of imperfect observations, which are generated by an observation model. Additionally, recognizing when a question admits no warranted answer and abstaining is treated as a first-class scored outcome. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding.","url_abs":"https://arxiv.org/abs/2607.08093","url_pdf":"https://arxiv.org/pdf/2607.08093","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.08093"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/andleb/causalds","reach":null}],"summary":{"ran":2,"ran_draft_wrong":13,"ran_honours":1,"unverified":17},"by_repo_kind":{"found_in_text":{"samples":33,"ran":16,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6fdf6553ceeed2b5","entry":"ExamComposition","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6fdf6553ceeed2b5"}},{"code_sha256_prefix":"e0d77746002a838d","entry":"TaskType","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0d77746002a838d"}},{"code_sha256_prefix":"471a02e4534852ff","entry":"_apply_obs_tilt","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"471a02e4534852ff"}},{"code_sha256_prefix":"7eb5370f70e56a33","entry":"_portable_repo_path","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7eb5370f70e56a33"}},{"code_sha256_prefix":"25b539302a57ef8b","entry":"_read_scene_structure_label","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25b539302a57ef8b"}},{"code_sha256_prefix":"cd7aca1869f92036","entry":"_resolve_family_weights","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cd7aca1869f92036"}},{"code_sha256_prefix":"cedc4f415bde2b77","entry":"_resolve_obs_weights","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cedc4f415bde2b77"}},{"code_sha256_prefix":"bfc09c3b80e6fe08","entry":"_scene_variant_dir","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bfc09c3b80e6fe08"}},{"code_sha256_prefix":"3d6d99900a05a2fd","entry":"_weighted_choice_from_available","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3d6d99900a05a2fd"}},{"code_sha256_prefix":"eae3e3a4f970a69c","entry":"list_scene_variants","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eae3e3a4f970a69c"}},{"code_sha256_prefix":"fbf133530644d05a","entry":"list_scenes","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fbf133530644d05a"}},{"code_sha256_prefix":"234aa5fbbbf6b308","entry":"load_scene_public","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"234aa5fbbbf6b308"}},{"code_sha256_prefix":"263573ac913eebcf","entry":"parse_outcome_type","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"263573ac913eebcf"}},{"code_sha256_prefix":"0986fc61d9da1545","entry":"parse_rung","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0986fc61d9da1545"}},{"code_sha256_prefix":"c7f4edee0b323065","entry":"parse_task_type","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7f4edee0b323065"}},{"code_sha256_prefix":"0a80820cffeef210","entry":"resolve_structural_label","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0a80820cffeef210"}},{"code_sha256_prefix":"0188381c0a0c1981","entry":"Exam","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0188381c0a0c1981"}},{"code_sha256_prefix":"ae97603a464ef027","entry":"ExamItem","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ae97603a464ef027"}},{"code_sha256_prefix":"f8896f6aa98efe68","entry":"OutcomeType","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8896f6aa98efe68"}},{"code_sha256_prefix":"010c920e7dcd9a49","entry":"OutputVariant","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"010c920e7dcd9a49"}},{"code_sha256_prefix":"51a4180de1afbc91","entry":"Rung","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51a4180de1afbc91"}},{"code_sha256_prefix":"fa486f10ca92e91f","entry":"TaskInputMode","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa486f10ca92e91f"}},{"code_sha256_prefix":"e5dd3c76df618a46","entry":"_apply_output_variant_tilt","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e5dd3c76df618a46"}},{"code_sha256_prefix":"321ae7d5014ac05c","entry":"_apply_rung_tilt","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"321ae7d5014ac05c"}},{"code_sha256_prefix":"8e248affd3e201fe","entry":"_assign_opaque_answer_file_stems","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8e248affd3e201fe"}},{"code_sha256_prefix":"e795869ed09af8ac","entry":"_normalize_composition","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e795869ed09af8ac"}},{"code_sha256_prefix":"26c8d8b2a34e4fbc","entry":"_resolve_variant_weights","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"26c8d8b2a34e4fbc"}},{"code_sha256_prefix":"8b6c01925666d449","entry":"build_exam","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b6c01925666d449"}},{"code_sha256_prefix":"4c28f9830b3a77fe","entry":"get_output_variant_difficulty_order","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c28f9830b3a77fe"}},{"code_sha256_prefix":"cffe0153b32ba4e9","entry":"get_task_input_mode","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cffe0153b32ba4e9"}},{"code_sha256_prefix":"c1f83096d08ad8bb","entry":"infer_rung_from_task_type","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c1f83096d08ad8bb"}},{"code_sha256_prefix":"9112deda4fa73ebe","entry":"normalize_task_fields","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9112deda4fa73ebe"}},{"code_sha256_prefix":"e709ab15df436195","entry":"parse_output_variant","repo":"andleb/causalds","repo_kind":"found_in_text","path":"causalds/exam_builder.py","file_url":"https://github.com/andleb/causalds/blob/HEAD/causalds/exam_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e709ab15df436195"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}