{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/model-free-and-model-based-policy-evaluation","title":"Model-Free and Model-Based Policy Evaluation when Causality is Uncertain","arxiv_id":"2204.00956","date":"2022-04-02","proceeding":null,"authors":["David Bruns-Smith"],"abstract":"When decision-makers can directly intervene, policy evaluation algorithms give valid causal estimates. In off-policy evaluation (OPE), there may exist unobserved variables that both impact the dynamics and are used by the unknown behavior policy. These \"confounders\" will introduce spurious correlations and naive estimates for a new policy will be biased. We develop worst-case bounds to assess sensitivity to these unobserved confounders in finite horizons when confounders are drawn iid each period. We demonstrate that a model-based approach with robust MDPs gives sharper lower bounds by exploiting domain knowledge about the dynamics. Finally, we show that when unobserved confounders are persistent over time, OPE is far more difficult and existing techniques produce extremely conservative bounds.","url_abs":"https://arxiv.org/abs/2204.00956v1","url_pdf":"https://arxiv.org/pdf/2204.00956v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"model-free-and-model-based-policy-evaluation","repo_url":"https://github.com/hetankevin/mdpmix","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"off-policy-evaluation","task_name":"Off-policy evaluation"},{"task_slug":"sensitivity","task_name":"Sensitivity"},{"task_slug":"model","task_name":"model"},{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.00956","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.00956"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hetankevin/mdpmix","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"listed":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"ad854d37b2ca9f3d","entry":"confound_V","repo":"hetankevin/mdpmix","repo_kind":"listed","path":"core/confound_env.py","file_url":"https://github.com/hetankevin/mdpmix/blob/HEAD/core/confound_env.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad854d37b2ca9f3d"}},{"code_sha256_prefix":"3fc547e23d654c96","entry":"getAcc","repo":"hetankevin/mdpmix","repo_kind":"listed","path":"mcmix/clustering.py","file_url":"https://github.com/hetankevin/mdpmix/blob/HEAD/mcmix/clustering.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3fc547e23d654c96"}},{"code_sha256_prefix":"b2cebbee44b630b2","entry":"rand_pi_val","repo":"hetankevin/mdpmix","repo_kind":"listed","path":"core/confound_env.py","file_url":"https://github.com/hetankevin/mdpmix/blob/HEAD/core/confound_env.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2cebbee44b630b2"}},{"code_sha256_prefix":"fa3adbcba7af3a0a","entry":"R_confound","repo":"hetankevin/mdpmix","repo_kind":"listed","path":"core/confound_env.py","file_url":"https://github.com/hetankevin/mdpmix/blob/HEAD/core/confound_env.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa3adbcba7af3a0a"}},{"code_sha256_prefix":"01a35169781ec8a6","entry":"getClusters","repo":"hetankevin/mdpmix","repo_kind":"listed","path":"mcmix/clustering.py","file_url":"https://github.com/hetankevin/mdpmix/blob/HEAD/mcmix/clustering.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01a35169781ec8a6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}