{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-courage-to-stop-overcoming-sunk-cost","title":"The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning","arxiv_id":"2506.13672","date":"2025-06-16","proceeding":null,"authors":["Jiashun Liu","Johan Obando-Ceron","Pablo Samuel Castro","Aaron Courville","Ling Pan"],"abstract":"Off-policy deep reinforcement learning (RL) typically leverages replay buffers for reusing past experiences during learning. This can help improve sample efficiency when the collected data is informative and aligned with the learning objectives; when that is not the case, it can have the effect of \"polluting\" the replay buffer with data which can exacerbate optimization challenges in addition to wasting environment interactions due to wasteful sampling. We argue that sampling these uninformative and wasteful transitions can be avoided by addressing the sunk cost fallacy, which, in the context of deep RL, is the tendency towards continuing an episode until termination. To address this, we propose learn to stop (LEAST), a lightweight mechanism that enables strategic early episode termination based on Q-value and gradient statistics, which helps agents recognize when to terminate unproductive episodes early. We demonstrate that our method improves learning efficiency on a variety of RL algorithms, evaluated on both the MuJoCo and DeepMind Control Suite benchmarks.","url_abs":"https://arxiv.org/abs/2506.13672v1","url_pdf":"https://arxiv.org/pdf/2506.13672v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.13672","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.13672"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sfujim/TD3","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/tyq1024/RLx2","reach":{"status":"ok"}}],"summary":{"unverified":4},"by_repo_kind":{"found_in_text":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"9e421ab3e0bca41f","entry":"get_W","repo":"tyq1024/RLx2","repo_kind":"found_in_text","path":"DST/utils.py","file_url":"https://github.com/tyq1024/RLx2/blob/HEAD/DST/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e421ab3e0bca41f"}},{"code_sha256_prefix":"cb21668945605238","entry":"get_weighted_layers","repo":"tyq1024/RLx2","repo_kind":"found_in_text","path":"DST/utils.py","file_url":"https://github.com/tyq1024/RLx2/blob/HEAD/DST/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb21668945605238"}},{"code_sha256_prefix":"74b0f54c764199f6","entry":"show_sparsity","repo":"tyq1024/RLx2","repo_kind":"found_in_text","path":"DST/utils.py","file_url":"https://github.com/tyq1024/RLx2/blob/HEAD/DST/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74b0f54c764199f6"}},{"code_sha256_prefix":"05ca958e48ca4299","entry":"sparse_set","repo":"tyq1024/RLx2","repo_kind":"found_in_text","path":"DST/DST_Scheduler.py","file_url":"https://github.com/tyq1024/RLx2/blob/HEAD/DST/DST_Scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"05ca958e48ca4299"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}