{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/finding-counterfactually-optimal-action","title":"Finding Counterfactually Optimal Action Sequences in Continuous State Spaces","arxiv_id":"2306.03929","date":"2023-06-06","proceeding":"NeurIPS 2023 11","authors":["Stratis Tsirtsis","Manuel Gomez-Rodriguez"],"abstract":"Whenever a clinician reflects on the efficacy of a sequence of treatment decisions for a patient, they may try to identify critical time steps where, had they made different decisions, the patient's health would have improved. While recent methods at the intersection of causal inference and reinforcement learning promise to aid human experts, as the clinician above, to retrospectively analyze sequential decision making processes, they have focused on environments with finitely many discrete states. However, in many practical applications, the state of the environment is inherently continuous in nature. In this paper, we aim to fill this gap. We start by formally characterizing a sequence of discrete actions and continuous states using finite horizon Markov decision processes and a broad class of bijective structural causal models. Building upon this characterization, we formalize the problem of finding counterfactually optimal action sequences and show that, in general, we cannot expect to solve it in polynomial time. Then, we develop a search method based on the $A^*$ algorithm that, under a natural form of Lipschitz continuity of the environment's dynamics, is guaranteed to return the optimal solution to the problem. Experiments on real clinical data show that our method is very efficient in practice, and it has the potential to offer interesting insights for sequential decision making tasks.","url_abs":"https://arxiv.org/abs/2306.03929v2","url_pdf":"https://arxiv.org/pdf/2306.03929v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"finding-counterfactually-optimal-action","repo_url":"https://github.com/networks-learning/counterfactual-continuous-mdp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"causal-inference","task_name":"Causal Inference"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"sequential-decision-making","task_name":"Sequential Decision Making"}],"methods":[{"method_slug":"causal-inference","method_name":"Causal inference"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.03929","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.03929"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/networks-learning/counterfactual-continuous-mdp","reach":null}],"summary":{"ran":3,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"9edad3fb99ec2891","entry":"CustomSpectralLinear","repo":"networks-learning/counterfactual-continuous-mdp","repo_kind":"official","path":"src/mimic_mdp.py","file_url":"https://github.com/networks-learning/counterfactual-continuous-mdp/blob/HEAD/src/mimic_mdp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9edad3fb99ec2891"}},{"code_sha256_prefix":"1900a71ed22526de","entry":"MultiplyConstantLayer","repo":"networks-learning/counterfactual-continuous-mdp","repo_kind":"official","path":"src/mimic_mdp.py","file_url":"https://github.com/networks-learning/counterfactual-continuous-mdp/blob/HEAD/src/mimic_mdp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1900a71ed22526de"}},{"code_sha256_prefix":"4b1d9d59ff1036c5","entry":"ParamMultivariateNormal","repo":"networks-learning/counterfactual-continuous-mdp","repo_kind":"official","path":"src/mimic_mdp.py","file_url":"https://github.com/networks-learning/counterfactual-continuous-mdp/blob/HEAD/src/mimic_mdp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b1d9d59ff1036c5"}},{"code_sha256_prefix":"4d6adcbe6f99381c","entry":"MimicMDP","repo":"networks-learning/counterfactual-continuous-mdp","repo_kind":"official","path":"src/mimic_mdp.py","file_url":"https://github.com/networks-learning/counterfactual-continuous-mdp/blob/HEAD/src/mimic_mdp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d6adcbe6f99381c"}},{"code_sha256_prefix":"e159ef0acd4bc71c","entry":"NeuralRegressor","repo":"networks-learning/counterfactual-continuous-mdp","repo_kind":"official","path":"src/mimic_mdp.py","file_url":"https://github.com/networks-learning/counterfactual-continuous-mdp/blob/HEAD/src/mimic_mdp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e159ef0acd4bc71c"}},{"code_sha256_prefix":"2b8907f345f3a4a9","entry":"Node","repo":"networks-learning/counterfactual-continuous-mdp","repo_kind":"official","path":"src/mimic_mdp.py","file_url":"https://github.com/networks-learning/counterfactual-continuous-mdp/blob/HEAD/src/mimic_mdp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b8907f345f3a4a9"}},{"code_sha256_prefix":"21b964db8b70436e","entry":"SCM","repo":"networks-learning/counterfactual-continuous-mdp","repo_kind":"official","path":"src/mimic_mdp.py","file_url":"https://github.com/networks-learning/counterfactual-continuous-mdp/blob/HEAD/src/mimic_mdp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"21b964db8b70436e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}