{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/implicit-distributional-reinforcement","title":"Implicit Distributional Reinforcement Learning","arxiv_id":"2007.06159","date":"2020-07-13","proceeding":"NeurIPS 2020 12","authors":["Yuguang Yue","Zhendong Wang","Mingyuan Zhou"],"abstract":"To improve the sample efficiency of policy-gradient based reinforcement learning algorithms, we propose implicit distributional actor-critic (IDAC) that consists of a distributional critic, built on two deep generator networks (DGNs), and a semi-implicit actor (SIA), powered by a flexible policy distribution. We adopt a distributional perspective on the discounted cumulative return and model it with a state-action-dependent implicit distribution, which is approximated by the DGNs that take state-action pairs and random noises as their input. Moreover, we use the SIA to provide a semi-implicit policy distribution, which mixes the policy parameters with a reparameterizable distribution that is not constrained by an analytic density function. In this way, the policy's marginal distribution is implicit, providing the potential to model complex properties such as covariance structure and skewness, but its parameter and entropy can still be estimated. We incorporate these features with an off-policy algorithm framework to solve problems with continuous action space and compare IDAC with state-of-the-art algorithms on representative OpenAI Gym environments. We observe that IDAC outperforms these baselines in most tasks. Python code is provided.","url_abs":"https://arxiv.org/abs/2007.06159v2","url_pdf":"https://arxiv.org/pdf/2007.06159v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"implicit-distributional-reinforcement","repo_url":"https://github.com/zhougroup/IDAC","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"implicit-distributional-reinforcement","repo_url":"https://github.com/SeungeonBaek/continuous-agents-test","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"implicit-distributional-reinforcement","repo_url":"https://github.com/yuguangyue/Implicit_Distributional_RL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"distributional-reinforcement-learning","task_name":"Distributional Reinforcement Learning"},{"task_slug":"openai-gym","task_name":"OpenAI Gym"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2007.06159","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2007.06159"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/hill-a/stable-baselines","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuguangyue/Implicit_Distributional_RL","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhougroup/IDAC","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SeungeonBaek/continuous-agents-test","reach":null}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"16606174c6ce3415","entry":"D_Critic","repo":"zhougroup/IDAC","repo_kind":"official","path":"agents/idac.py","file_url":"https://github.com/zhougroup/IDAC/blob/HEAD/agents/idac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16606174c6ce3415"}},{"code_sha256_prefix":"d595cdaaa075b310","entry":"G_Actor","repo":"zhougroup/IDAC","repo_kind":"official","path":"agents/idac.py","file_url":"https://github.com/zhougroup/IDAC/blob/HEAD/agents/idac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d595cdaaa075b310"}},{"code_sha256_prefix":"f31bac0ea9e377cf","entry":"Implicit_Actor","repo":"zhougroup/IDAC","repo_kind":"official","path":"agents/idac.py","file_url":"https://github.com/zhougroup/IDAC/blob/HEAD/agents/idac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f31bac0ea9e377cf"}},{"code_sha256_prefix":"578423b4a2bb1cf9","entry":"TanhNormal","repo":"zhougroup/IDAC","repo_kind":"official","path":"agents/idac.py","file_url":"https://github.com/zhougroup/IDAC/blob/HEAD/agents/idac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"578423b4a2bb1cf9"}},{"code_sha256_prefix":"2678e0fdb62a4cf9","entry":"DistCritic","repo":"SeungeonBaek/continuous-agents-test","repo_kind":"listed","path":"agents/IDAC.py","file_url":"https://github.com/SeungeonBaek/continuous-agents-test/blob/HEAD/agents/IDAC.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2678e0fdb62a4cf9"}},{"code_sha256_prefix":"3fa2247f371c4678","entry":"IDAC","repo":"zhougroup/IDAC","repo_kind":"official","path":"agents/idac.py","file_url":"https://github.com/zhougroup/IDAC/blob/HEAD/agents/idac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3fa2247f371c4678"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}