{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-value-learning-really-the-main-bottleneck","title":"Is Value Learning Really the Main Bottleneck in Offline RL?","arxiv_id":"2406.09329","date":"2024-06-13","proceeding":null,"authors":["Seohong Park","Kevin Frans","Sergey Levine","Aviral Kumar"],"abstract":"While imitation learning requires access to high-quality data, offline reinforcement learning (RL) should, in principle, perform similarly or better with substantially lower data quality by using a value function. However, current results indicate that offline RL often performs worse than imitation learning, and it is often unclear what holds back the performance of offline RL. Motivated by this observation, we aim to understand the bottlenecks in current offline RL algorithms. While poor performance of offline RL is typically attributed to an imperfect value function, we ask: is the main bottleneck of offline RL indeed in learning the value function, or something else? To answer this question, we perform a systematic empirical study of (1) value learning, (2) policy extraction, and (3) policy generalization in offline RL problems, analyzing how these components affect performance. We make two surprising observations. First, we find that the choice of a policy extraction algorithm significantly affects the performance and scalability of offline RL, often more so than the value learning objective. For instance, we show that common value-weighted behavioral cloning objectives (e.g., AWR) do not fully leverage the learned value function, and switching to behavior-constrained policy gradient objectives (e.g., DDPG+BC) often leads to substantial improvements in performance and scalability. Second, we find that a big barrier to improving offline RL performance is often imperfect policy generalization on test-time states out of the support of the training data, rather than policy learning on in-distribution states. We then show that the use of suboptimal but high-coverage data or test-time policy training techniques can address this generalization issue in practice. Specifically, we propose two simple test-time policy improvement methods and show that these methods lead to better performance.","url_abs":"https://arxiv.org/abs/2406.09329v2","url_pdf":"https://arxiv.org/pdf/2406.09329v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-value-learning-really-the-main-bottleneck","repo_url":"https://github.com/seohongpark/ogbench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.09329","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.09329"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/seohongpark/ogbench","reach":null}],"summary":{"ran":7,"unverified":3},"by_repo_kind":{"listed":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9bc539476ffcf222","entry":"GCActor","repo":"seohongpark/ogbench","repo_kind":"listed","path":"impls/agents/hiql.py","file_url":"https://github.com/seohongpark/ogbench/blob/HEAD/impls/agents/hiql.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9bc539476ffcf222"}},{"code_sha256_prefix":"85c24c360661275c","entry":"GCEncoder","repo":"seohongpark/ogbench","repo_kind":"listed","path":"impls/agents/hiql.py","file_url":"https://github.com/seohongpark/ogbench/blob/HEAD/impls/agents/hiql.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85c24c360661275c"}},{"code_sha256_prefix":"1a0dd970d5292061","entry":"GCValue","repo":"seohongpark/ogbench","repo_kind":"listed","path":"impls/agents/hiql.py","file_url":"https://github.com/seohongpark/ogbench/blob/HEAD/impls/agents/hiql.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a0dd970d5292061"}},{"code_sha256_prefix":"55eac5ab51a6832f","entry":"Identity","repo":"seohongpark/ogbench","repo_kind":"listed","path":"impls/agents/hiql.py","file_url":"https://github.com/seohongpark/ogbench/blob/HEAD/impls/agents/hiql.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55eac5ab51a6832f"}},{"code_sha256_prefix":"503831beac2bf3d9","entry":"ImpalaEncoder","repo":"seohongpark/ogbench","repo_kind":"listed","path":"impls/agents/hiql.py","file_url":"https://github.com/seohongpark/ogbench/blob/HEAD/impls/agents/hiql.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"503831beac2bf3d9"}},{"code_sha256_prefix":"ccbb1f2bba20397b","entry":"MLP","repo":"seohongpark/ogbench","repo_kind":"listed","path":"impls/agents/hiql.py","file_url":"https://github.com/seohongpark/ogbench/blob/HEAD/impls/agents/hiql.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ccbb1f2bba20397b"}},{"code_sha256_prefix":"1b4a14ceaf967da5","entry":"ModuleDict","repo":"seohongpark/ogbench","repo_kind":"listed","path":"impls/agents/hiql.py","file_url":"https://github.com/seohongpark/ogbench/blob/HEAD/impls/agents/hiql.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b4a14ceaf967da5"}},{"code_sha256_prefix":"76e656aaad5ce115","entry":"GCDiscreteActor","repo":"seohongpark/ogbench","repo_kind":"listed","path":"impls/agents/hiql.py","file_url":"https://github.com/seohongpark/ogbench/blob/HEAD/impls/agents/hiql.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76e656aaad5ce115"}},{"code_sha256_prefix":"3c394762e2a85d6f","entry":"HIQLAgent","repo":"seohongpark/ogbench","repo_kind":"listed","path":"impls/agents/hiql.py","file_url":"https://github.com/seohongpark/ogbench/blob/HEAD/impls/agents/hiql.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c394762e2a85d6f"}},{"code_sha256_prefix":"2c53e405a504fd6d","entry":"LengthNormalize","repo":"seohongpark/ogbench","repo_kind":"listed","path":"impls/agents/hiql.py","file_url":"https://github.com/seohongpark/ogbench/blob/HEAD/impls/agents/hiql.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2c53e405a504fd6d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}