{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reinforcement-learning-in-dynamic-treatment","title":"Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination","arxiv_id":"2405.18556","date":"2024-05-28","proceeding":null,"authors":["Zhiyao Luo","Yangchen Pan","Peter Watkinson","Tingting Zhu"],"abstract":"In the rapidly changing healthcare landscape, the implementation of offline reinforcement learning (RL) in dynamic treatment regimes (DTRs) presents a mix of unprecedented opportunities and challenges. This position paper offers a critical examination of the current status of offline RL in the context of DTRs. We argue for a reassessment of applying RL in DTRs, citing concerns such as inconsistent and potentially inconclusive evaluation metrics, the absence of naive and supervised learning baselines, and the diverse choice of RL formulation in existing research. Through a case study with more than 17,000 evaluation experiments using a publicly available Sepsis dataset, we demonstrate that the performance of RL algorithms can significantly vary with changes in evaluation metrics and Markov Decision Process (MDP) formulations. Surprisingly, it is observed that in some instances, RL algorithms can be surpassed by random baselines subjected to policy evaluation methods and reward design. This calls for more careful policy evaluation and algorithm development in future DTR works. Additionally, we discussed potential enhancements toward more reliable development of RL-based dynamic treatment regimes and invited further discussion within the community. Code is available at https://github.com/GilesLuo/ReassessDTR.","url_abs":"https://arxiv.org/abs/2405.18556v2","url_pdf":"https://arxiv.org/pdf/2405.18556v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reinforcement-learning-in-dynamic-treatment","repo_url":"https://github.com/gilesluo/reassessdtr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2405.18556","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.18556"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/GilesLuo/ReassessDTR","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gilesluo/reassessdtr","reach":{"status":"ok"}}],"summary":{"ran":8,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"7e6f27066c11765f","entry":"get_APACHE2_score","repo":"GilesLuo/ReassessDTR","repo_kind":"official","path":"DTRGym/icu_scoring.py","file_url":"https://github.com/GilesLuo/ReassessDTR/blob/HEAD/DTRGym/icu_scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e6f27066c11765f"}},{"code_sha256_prefix":"1346e0d1af9abcc9","entry":"get_NEWS2_score","repo":"GilesLuo/ReassessDTR","repo_kind":"official","path":"DTRGym/icu_scoring.py","file_url":"https://github.com/GilesLuo/ReassessDTR/blob/HEAD/DTRGym/icu_scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1346e0d1af9abcc9"}},{"code_sha256_prefix":"130230aa11c3c8cb","entry":"get_common_hparams","repo":"GilesLuo/ReassessDTR","repo_kind":"official","path":"HD4RL/core/base_hparams.py","file_url":"https://github.com/GilesLuo/ReassessDTR/blob/HEAD/HD4RL/core/base_hparams.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"130230aa11c3c8cb"}},{"code_sha256_prefix":"687d05a4f65717ae","entry":"get_neighbors","repo":"GilesLuo/ReassessDTR","repo_kind":"official","path":"DTRGym/utils.py","file_url":"https://github.com/GilesLuo/ReassessDTR/blob/HEAD/DTRGym/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"687d05a4f65717ae"}},{"code_sha256_prefix":"d3d182a5017557ac","entry":"get_neighbors_indices","repo":"GilesLuo/ReassessDTR","repo_kind":"official","path":"DTRGym/utils.py","file_url":"https://github.com/GilesLuo/ReassessDTR/blob/HEAD/DTRGym/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3d182a5017557ac"}},{"code_sha256_prefix":"b42cab51e3e9b012","entry":"in_range","repo":"GilesLuo/ReassessDTR","repo_kind":"official","path":"DTRGym/icu_scoring.py","file_url":"https://github.com/GilesLuo/ReassessDTR/blob/HEAD/DTRGym/icu_scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b42cab51e3e9b012"}},{"code_sha256_prefix":"a995b9f7d9efe810","entry":"is_square_number","repo":"GilesLuo/ReassessDTR","repo_kind":"official","path":"DTRGym/utils.py","file_url":"https://github.com/GilesLuo/ReassessDTR/blob/HEAD/DTRGym/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a995b9f7d9efe810"}},{"code_sha256_prefix":"32cdbcca857a4d35","entry":"uniform_random","repo":"GilesLuo/ReassessDTR","repo_kind":"official","path":"DTRGym/base.py","file_url":"https://github.com/GilesLuo/ReassessDTR/blob/HEAD/DTRGym/base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32cdbcca857a4d35"}},{"code_sha256_prefix":"92c21a9c487dd853","entry":"policy_load","repo":"GilesLuo/ReassessDTR","repo_kind":"official","path":"HD4RL/core/helper_fn.py","file_url":"https://github.com/GilesLuo/ReassessDTR/blob/HEAD/HD4RL/core/helper_fn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92c21a9c487dd853"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}