{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-14779","title":"Peng's Q($λ$) for Conservative Value Estimation in Offline Reinforcement Learning","arxiv_id":"2605.14779","date":"2026-05-14","proceeding":null,"authors":["Byeongchan Kim","Min-hwan Oh"],"abstract":"We propose a model-free offline multi-step reinforcement learning (RL) algorithm, Conservative Peng's Q($λ$) (CPQL). Our algorithm adapts the Peng's Q($λ$) (PQL) operator for conservative value estimation as an alternative to the Bellman operator. To the best of our knowledge, this is the first work in offline RL to theoretically and empirically demonstrate the effectiveness of conservative value estimation with a \\textit{multi-step} operator by fully leveraging offline trajectories. The fixed point of the PQL operator in offline RL lies closer to the value function of the behavior policy, thereby naturally inducing implicit behavior regularization. CPQL simultaneously mitigates over-pessimistic value estimation, achieves performance greater than (or equal to) that of the behavior policy, and provides near-optimal performance guarantees -- a milestone that previous conservative approaches could not achieve. Extensive numerical experiments on the D4RL benchmark demonstrate that CPQL consistently and significantly outperforms existing offline single-step baselines. In addition to the contributions of CPQL in offline RL, our proposed method also contributes to the offline-to-online learning framework. Using the Q-function pre-trained by CPQL in offline settings enables the online PQL agent to avoid the performance drop typically observed at the start of fine-tuning and to attain robust performance improvements. Our code is available at https://github.com/oh-lab/CPQL.","url_abs":"https://arxiv.org/abs/2605.14779","url_pdf":"https://arxiv.org/pdf/2605.14779","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.14779","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.14779"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/2023AnnonymousAuthor/csve","reach":{"status":"ok","spdx":"MIT"}},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/tinkoff-ai/CORL","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ohlab/CPQL","reach":{"status":"gone","observed_at":"2026-09-16","how":"tree_404+repo_404"}},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/hyeon1996/EPQ","reach":null}],"summary":{"ran":7,"ran_honours":1,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"fadf5f762ff04cd6","entry":"create_stats_ordered_dict","repo":"hyeon1996/EPQ","repo_kind":"found_in_text","path":"rlkit/core/eval_util.py","file_url":"https://github.com/hyeon1996/EPQ/blob/HEAD/rlkit/core/eval_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fadf5f762ff04cd6"}},{"code_sha256_prefix":"a5d10a0b704c8db7","entry":"get_average_returns","repo":"hyeon1996/EPQ","repo_kind":"found_in_text","path":"rlkit/core/eval_util.py","file_url":"https://github.com/hyeon1996/EPQ/blob/HEAD/rlkit/core/eval_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5d10a0b704c8db7"}},{"code_sha256_prefix":"9b9a51e3e3be7a6e","entry":"get_flat_params","repo":"hyeon1996/EPQ","repo_kind":"found_in_text","path":"rlkit/core/batch_rl_algorithm.py","file_url":"https://github.com/hyeon1996/EPQ/blob/HEAD/rlkit/core/batch_rl_algorithm.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b9a51e3e3be7a6e"}},{"code_sha256_prefix":"9910e2fc297f8665","entry":"identity","repo":"hyeon1996/EPQ","repo_kind":"found_in_text","path":"rlkit/torch/sac/epq_trainer.py","file_url":"https://github.com/hyeon1996/EPQ/blob/HEAD/rlkit/torch/sac/epq_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9910e2fc297f8665"}},{"code_sha256_prefix":"518461d2aec0dfde","entry":"load_state_dict_data","repo":"hyeon1996/EPQ","repo_kind":"found_in_text","path":"behavior_cloning.py","file_url":"https://github.com/hyeon1996/EPQ/blob/HEAD/behavior_cloning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"518461d2aec0dfde"}},{"code_sha256_prefix":"148690949a91e459","entry":"remove_dup","repo":"hyeon1996/EPQ","repo_kind":"found_in_text","path":"clustering.py","file_url":"https://github.com/hyeon1996/EPQ/blob/HEAD/clustering.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"148690949a91e459"}},{"code_sha256_prefix":"5cb0c55f58279f59","entry":"simple_separated_format","repo":"hyeon1996/EPQ","repo_kind":"found_in_text","path":"rlkit/core/tabulate.py","file_url":"https://github.com/hyeon1996/EPQ/blob/HEAD/rlkit/core/tabulate.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5cb0c55f58279f59"}},{"code_sha256_prefix":"981dd701152da17c","entry":"str2bool","repo":"hyeon1996/EPQ","repo_kind":"found_in_text","path":"epq_main.py","file_url":"https://github.com/hyeon1996/EPQ/blob/HEAD/epq_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"981dd701152da17c"}},{"code_sha256_prefix":"ed641ef1d8cfdb43","entry":"load_state_dict_data","repo":"hyeon1996/EPQ","repo_kind":"found_in_text","path":"epq_main.py","file_url":"https://github.com/hyeon1996/EPQ/blob/HEAD/epq_main.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ed641ef1d8cfdb43"}},{"code_sha256_prefix":"55a35c24613f6943","entry":"nearest_neighborhood_cluster","repo":"hyeon1996/EPQ","repo_kind":"found_in_text","path":"clustering.py","file_url":"https://github.com/hyeon1996/EPQ/blob/HEAD/clustering.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55a35c24613f6943"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}