{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/predictive-performance-comparison-of-decision","title":"Predictive Performance Comparison of Decision Policies Under Confounding","arxiv_id":"2404.00848","date":"2024-04-01","proceeding":null,"authors":["Luke Guerdan","Amanda Coston","Kenneth Holstein","Zhiwei Steven Wu"],"abstract":"Predictive models are often introduced to decision-making tasks under the rationale that they improve performance over an existing decision-making policy. However, it is challenging to compare predictive performance against an existing decision-making policy that is generally under-specified and dependent on unobservable factors. These sources of uncertainty are often addressed in practice by making strong assumptions about the data-generating mechanism. In this work, we propose a method to compare the predictive performance of decision policies under a variety of modern identification approaches from the causal inference and off-policy evaluation literatures (e.g., instrumental variable, marginal sensitivity model, proximal variable). Key to our method is the insight that there are regions of uncertainty that we can safely ignore in the policy comparison. We develop a practical approach for finite-sample estimation of regret intervals under no assumptions on the parametric form of the status quo policy. We verify our framework theoretically and via synthetic data experiments. We conclude with a real-world application using our framework to support a pre-deployment evaluation of a proposed modification to a healthcare enrollment policy.","url_abs":"https://arxiv.org/abs/2404.00848v2","url_pdf":"https://arxiv.org/pdf/2404.00848v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"predictive-performance-comparison-of-decision","repo_url":"https://github.com/lguerdan/icml24_predictive_performance_comparison_dps","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"causal-inference","task_name":"Causal Inference"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"off-policy-evaluation","task_name":"Off-policy evaluation"}],"methods":[{"method_slug":"causal-inference","method_name":"Causal inference"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.00848","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.00848"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lguerdan/icml24_predictive_performance_comparison_dps","reach":null}],"summary":{"ran_fixture":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"8da9ba7c23f0e97f","entry":"delta_bounds","repo":"lguerdan/icml24_predictive_performance_comparison_dps","repo_kind":"official","path":"bounds.py","file_url":"https://github.com/lguerdan/icml24_predictive_performance_comparison_dps/blob/HEAD/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8da9ba7c23f0e97f"}},{"code_sha256_prefix":"fe6c1a20554a427e","entry":"oracle_regret","repo":"lguerdan/icml24_predictive_performance_comparison_dps","repo_kind":"official","path":"bounds.py","file_url":"https://github.com/lguerdan/icml24_predictive_performance_comparison_dps/blob/HEAD/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe6c1a20554a427e"}},{"code_sha256_prefix":"d7540d613a473bb6","entry":"standard_bounds","repo":"lguerdan/icml24_predictive_performance_comparison_dps","repo_kind":"official","path":"bounds.py","file_url":"https://github.com/lguerdan/icml24_predictive_performance_comparison_dps/blob/HEAD/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d7540d613a473bb6"}},{"code_sha256_prefix":"3d538d5232e8d9eb","entry":"get_bounds","repo":"lguerdan/icml24_predictive_performance_comparison_dps","repo_kind":"official","path":"bounds.py","file_url":"https://github.com/lguerdan/icml24_predictive_performance_comparison_dps/blob/HEAD/bounds.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d538d5232e8d9eb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}