{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/variational-delayed-policy-optimization","title":"Variational Delayed Policy Optimization","arxiv_id":"2405.14226","date":"2024-05-23","proceeding":null,"authors":["Qingyuan Wu","Simon Sinong Zhan","YiXuan Wang","Yuhui Wang","Chung-Wei Lin","Chen Lv","Qi Zhu","Chao Huang"],"abstract":"In environments with delayed observation, state augmentation by including actions within the delay window is adopted to retrieve Markovian property to enable reinforcement learning (RL). However, state-of-the-art (SOTA) RL techniques with Temporal-Difference (TD) learning frameworks often suffer from learning inefficiency, due to the significant expansion of the augmented state space with the delay. To improve learning efficiency without sacrificing performance, this work introduces a novel framework called Variational Delayed Policy Optimization (VDPO), which reformulates delayed RL as a variational inference problem. This problem is further modelled as a two-step iterative optimization problem, where the first step is TD learning in the delay-free environment with a small state space, and the second step is behaviour cloning which can be addressed much more efficiently than TD learning. We not only provide a theoretical analysis of VDPO in terms of sample complexity and performance, but also empirically demonstrate that VDPO can achieve consistent performance with SOTA methods, with a significant enhancement of sample efficiency (approximately 50\\% less amount of samples) in the MuJoCo benchmark.","url_abs":"https://arxiv.org/abs/2405.14226v2","url_pdf":"https://arxiv.org/pdf/2405.14226v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"variational-delayed-policy-optimization","repo_url":"https://github.com/qingyuanwunothing/vdpo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"variational-inference","task_name":"Variational Inference"}],"methods":[{"method_slug":"variational-inference","method_name":"Variational Inference"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.14226","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.14226"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/QingyuanWuNothing/VDPO","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qingyuanwunothing/vdpo","reach":{"status":"ok"}}],"summary":{"ran":4,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"8d481d190b83d758","entry":"find_specific_exp_tag","repo":"QingyuanWuNothing/VDPO","repo_kind":"official","path":"utils.py","file_url":"https://github.com/QingyuanWuNothing/VDPO/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d481d190b83d758"}},{"code_sha256_prefix":"9af8fadcf673cf17","entry":"kl_divergence","repo":"qingyuanwunothing/vdpo","repo_kind":"official","path":"VDPO.py","file_url":"https://github.com/qingyuanwunothing/vdpo/blob/HEAD/VDPO.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9af8fadcf673cf17"}},{"code_sha256_prefix":"beb42d440ccf2925","entry":"layer_init","repo":"QingyuanWuNothing/VDPO","repo_kind":"official","path":"nn.py","file_url":"https://github.com/QingyuanWuNothing/VDPO/blob/HEAD/nn.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"beb42d440ccf2925"}},{"code_sha256_prefix":"c21517332c327b92","entry":"make_anneal_lr","repo":"QingyuanWuNothing/VDPO","repo_kind":"official","path":"utils.py","file_url":"https://github.com/QingyuanWuNothing/VDPO/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c21517332c327b92"}},{"code_sha256_prefix":"99b67773f8dfb8b3","entry":"make_delayed_mujoco_env","repo":"QingyuanWuNothing/VDPO","repo_kind":"official","path":"make_env.py","file_url":"https://github.com/QingyuanWuNothing/VDPO/blob/HEAD/make_env.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99b67773f8dfb8b3"}},{"code_sha256_prefix":"bdf1781d7265fc58","entry":"make_linear_schedule","repo":"QingyuanWuNothing/VDPO","repo_kind":"official","path":"utils.py","file_url":"https://github.com/QingyuanWuNothing/VDPO/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bdf1781d7265fc58"}},{"code_sha256_prefix":"308cb4e474e256a9","entry":"make_mujoco_env","repo":"QingyuanWuNothing/VDPO","repo_kind":"official","path":"make_env.py","file_url":"https://github.com/QingyuanWuNothing/VDPO/blob/HEAD/make_env.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"308cb4e474e256a9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}