{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tiered-reinforcement-learning-pessimism-in","title":"Tiered Reinforcement Learning: Pessimism in the Face of Uncertainty and Constant Regret","arxiv_id":"2205.12418","date":"2022-05-25","proceeding":null,"authors":["Jiawei Huang","Li Zhao","Tao Qin","Wei Chen","Nan Jiang","Tie-Yan Liu"],"abstract":"We propose a new learning framework that captures the tiered structure of many real-world user-interaction applications, where the users can be divided into two groups based on their different tolerance on exploration risks and should be treated separately. In this setting, we simultaneously maintain two policies $\\pi^{\\text{O}}$ and $\\pi^{\\text{E}}$: $\\pi^{\\text{O}}$ (\"O\" for \"online\") interacts with more risk-tolerant users from the first tier and minimizes regret by balancing exploration and exploitation as usual, while $\\pi^{\\text{E}}$ (\"E\" for \"exploit\") exclusively focuses on exploitation for risk-averse users from the second tier utilizing the data collected so far. An important question is whether such a separation yields advantages over the standard online setting (i.e., $\\pi^{\\text{E}}=\\pi^{\\text{O}}$) for the risk-averse users. We individually consider the gap-independent vs.~gap-dependent settings. For the former, we prove that the separation is indeed not beneficial from a minimax perspective. For the latter, we show that if choosing Pessimistic Value Iteration as the exploitation algorithm to produce $\\pi^{\\text{E}}$, we can achieve a constant regret for risk-averse users independent of the number of episodes $K$, which is in sharp contrast to the $\\Omega(\\log K)$ regret for any online RL algorithms in the same setting, while the regret of $\\pi^{\\text{O}}$ (almost) maintains its online regret optimality and does not need to compromise for the success of $\\pi^{\\text{E}}$.","url_abs":"https://arxiv.org/abs/2205.12418v4","url_pdf":"https://arxiv.org/pdf/2205.12418v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tiered-reinforcement-learning-pessimism-in","repo_url":"https://github.com/jiaweihhuang/tiered-rl-experiments","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2205.12418","atlas_url":"https://app.syntology.ai/?focus=2205.12418","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.12418"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiaweihhuang/tiered-rl-experiments","reach":null}],"summary":{"ran":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"2e1f15ef32cf7d0a","entry":"Algorithm","repo":"jiaweihhuang/tiered-rl-experiments","repo_kind":"official","path":"Tabular_MDP.py","file_url":"https://github.com/jiaweihhuang/tiered-rl-experiments/blob/HEAD/Tabular_MDP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e1f15ef32cf7d0a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}