{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/carla-a-python-library-to-benchmark","title":"CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms","arxiv_id":"2108.00783","date":"2021-08-02","proceeding":null,"authors":["Martin Pawelczyk","Sascha Bielawski","Johannes van den Heuvel","Tobias Richter","Gjergji Kasneci"],"abstract":"Counterfactual explanations provide means for prescriptive model explanations by suggesting actionable feature changes (e.g., increase income) that allow individuals to achieve favorable outcomes in the future (e.g., insurance approval). Choosing an appropriate method is a crucial aspect for meaningful counterfactual explanations. As documented in recent reviews, there exists a quickly growing literature with available methods. Yet, in the absence of widely available opensource implementations, the decision in favor of certain models is primarily based on what is readily available. Going forward - to guarantee meaningful comparisons across explanation methods - we present CARLA (Counterfactual And Recourse LibrAry), a python library for benchmarking counterfactual explanation methods across both different data sets and different machine learning models. In summary, our work provides the following contributions: (i) an extensive benchmark of 11 popular counterfactual explanation methods, (ii) a benchmarking framework for research on future counterfactual explanation methods, and (iii) a standardized set of integrated evaluation measures and data sets for transparent and extensive comparisons of these methods. We have open-sourced CARLA and our experimental results on Github, making them available as competitive baselines. We welcome contributions from other research groups and practitioners.","url_abs":"https://arxiv.org/abs/2108.00783v1","url_pdf":"https://arxiv.org/pdf/2108.00783v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"carla-a-python-library-to-benchmark","repo_url":"https://github.com/indyfree/CARLA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"carla-a-python-library-to-benchmark","repo_url":"https://github.com/MartinPawel/ProbabilisticallyRobustRecourse","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"carla-a-python-library-to-benchmark","repo_url":"https://github.com/Orange-OpenSource/carla_croco_vcnet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"carla-a-python-library-to-benchmark","repo_url":"https://github.com/carla-recourse/CARLA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"counterfactual-explanation","task_name":"Counterfactual Explanation"},{"task_slug":null,"task_name":"counterfactual"}],"methods":[{"method_slug":"carla","method_name":"CARLA"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2108.00783","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2108.00783"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MartinPawel/ProbabilisticallyRobustRecourse","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/carla-recourse/CARLA","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/indyfree/CARLA","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Orange-OpenSource/carla_croco_vcnet","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5b72258632321738","entry":"get_models_home","repo":"indyfree/CARLA","repo_kind":"official","path":"carla/models/catalog/load_model.py","file_url":"https://github.com/indyfree/CARLA/blob/HEAD/carla/models/catalog/load_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b72258632321738"}},{"code_sha256_prefix":"142ef8036899df58","entry":"load_online_model","repo":"indyfree/CARLA","repo_kind":"official","path":"carla/models/catalog/load_model.py","file_url":"https://github.com/indyfree/CARLA/blob/HEAD/carla/models/catalog/load_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"142ef8036899df58"}},{"code_sha256_prefix":"a86c4a61b987d962","entry":"load_trained_model","repo":"indyfree/CARLA","repo_kind":"official","path":"carla/models/catalog/load_model.py","file_url":"https://github.com/indyfree/CARLA/blob/HEAD/carla/models/catalog/load_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a86c4a61b987d962"}},{"code_sha256_prefix":"454cbfa6263a3d56","entry":"logistic_function","repo":"indyfree/CARLA","repo_kind":"official","path":"carla/models/catalog/parse_xgboost.py","file_url":"https://github.com/indyfree/CARLA/blob/HEAD/carla/models/catalog/parse_xgboost.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"454cbfa6263a3d56"}},{"code_sha256_prefix":"bd9c836dd3c45c49","entry":"parse_booster","repo":"indyfree/CARLA","repo_kind":"official","path":"carla/models/catalog/parse_xgboost.py","file_url":"https://github.com/indyfree/CARLA/blob/HEAD/carla/models/catalog/parse_xgboost.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd9c836dd3c45c49"}},{"code_sha256_prefix":"5045a097ab96a95f","entry":"predict_label","repo":"indyfree/CARLA","repo_kind":"official","path":"carla/models/negative_instances/predict.py","file_url":"https://github.com/indyfree/CARLA/blob/HEAD/carla/models/negative_instances/predict.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5045a097ab96a95f"}},{"code_sha256_prefix":"693981f8240930fa","entry":"predict_negative_instances","repo":"indyfree/CARLA","repo_kind":"official","path":"carla/models/negative_instances/predict.py","file_url":"https://github.com/indyfree/CARLA/blob/HEAD/carla/models/negative_instances/predict.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"693981f8240930fa"}},{"code_sha256_prefix":"94cef7815945f3ca","entry":"remove_nans","repo":"indyfree/CARLA","repo_kind":"official","path":"carla/evaluation/process_nans.py","file_url":"https://github.com/indyfree/CARLA/blob/HEAD/carla/evaluation/process_nans.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94cef7815945f3ca"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}