{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/optimizing-for-the-future-in-non-stationary","title":"Optimizing for the Future in Non-Stationary MDPs","arxiv_id":"2005.08158","date":"2020-05-17","proceeding":"ICML 2020 1","authors":["Yash Chandak","Georgios Theocharous","Shiv Shankar","Martha White","Sridhar Mahadevan","Philip S. Thomas"],"abstract":"Most reinforcement learning methods are based upon the key assumption that the transition dynamics and reward functions are fixed, that is, the underlying Markov decision process is stationary. However, in many real-world applications, this assumption is violated, and using existing algorithms may result in a performance lag. To proactively search for a good future policy, we present a policy gradient algorithm that maximizes a forecast of future performance. This forecast is obtained by fitting a curve to the counter-factual estimates of policy performance over time, without explicitly modeling the underlying non-stationarity. The resulting algorithm amounts to a non-uniform reweighting of past data, and we observe that minimizing performance over some of the data from past episodes can be beneficial when searching for a policy that maximizes future performance. We show that our algorithm, called Prognosticator, is more robust to non-stationarity than two online adaptation techniques, on three simulated problems motivated by real-world applications.","url_abs":"https://arxiv.org/abs/2005.08158v4","url_pdf":"https://arxiv.org/pdf/2005.08158v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimizing-for-the-future-in-non-stationary","repo_url":"https://github.com/yashchandak/OptFuture_NSMDP","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2005.08158","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.08158"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yashchandak/OptFuture_NSMDP","reach":null}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"c678ff92bd3f8874","entry":"WLS","repo":"yashchandak/OptFuture_NSMDP","repo_kind":"official","path":"Src/Algorithms/ProWLS.py","file_url":"https://github.com/yashchandak/OptFuture_NSMDP/blob/HEAD/Src/Algorithms/ProWLS.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c678ff92bd3f8874"}},{"code_sha256_prefix":"dccc58ed9546ba0e","entry":"ProWLS","repo":"yashchandak/OptFuture_NSMDP","repo_kind":"official","path":"Src/Algorithms/ProWLS.py","file_url":"https://github.com/yashchandak/OptFuture_NSMDP/blob/HEAD/Src/Algorithms/ProWLS.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dccc58ed9546ba0e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}