{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cebab-estimating-the-causal-effects-of-real","title":"CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior","arxiv_id":"2205.14140","date":"2022-05-27","proceeding":null,"authors":["Eldar David Abraham","Karel D'Oosterlinck","Amir Feder","Yair Ori Gat","Atticus Geiger","Christopher Potts","Roi Reichart","Zhengxuan Wu"],"abstract":"The increasing size and complexity of modern ML systems has improved their predictive capabilities but made their behavior harder to explain. Many techniques for model explanation have been developed in response, but we lack clear criteria for assessing these techniques. In this paper, we cast model explanation as the causal inference problem of estimating causal effects of real-world concepts on the output behavior of ML models given actual input data. We introduce CEBaB, a new benchmark dataset for assessing concept-based explanation methods in Natural Language Processing (NLP). CEBaB consists of short restaurant reviews with human-generated counterfactual reviews in which an aspect (food, noise, ambiance, service) of the dining experience was modified. Original and counterfactual reviews are annotated with multiply-validated sentiment ratings at the aspect-level and review-level. The rich structure of CEBaB allows us to go beyond input features to study the effects of abstract, real-world concepts on model behavior. We use CEBaB to compare the quality of a range of concept-based explanation methods covering different assumptions and conceptions of the problem, and we seek to establish natural metrics for comparative assessments of these methods.","url_abs":"https://arxiv.org/abs/2205.14140v2","url_pdf":"https://arxiv.org/pdf/2205.14140v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cebab-estimating-the-causal-effects-of-real","repo_url":"https://github.com/cebabing/cebab","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"causal-inference","task_name":"Causal Inference"},{"task_slug":null,"task_name":"counterfactual"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.14140","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.14140"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cebabing/cebab","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"listed":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"b4296c90d4c0e2ce","entry":"to_ternary","repo":"cebabing/cebab","repo_kind":"listed","path":"code/eval_pipeline/explainers/causalm/utils/cebab_dataset.py","file_url":"https://github.com/cebabing/cebab/blob/HEAD/code/eval_pipeline/explainers/causalm/utils/cebab_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4296c90d4c0e2ce"}},{"code_sha256_prefix":"8f5d78c20151198b","entry":"process_aspect","repo":"cebabing/cebab","repo_kind":"listed","path":"code/eval_pipeline/explainers/causalm/utils/cebab_dataset.py","file_url":"https://github.com/cebabing/cebab/blob/HEAD/code/eval_pipeline/explainers/causalm/utils/cebab_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f5d78c20151198b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}