{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/munchausen-reinforcement-learning","title":"Munchausen Reinforcement Learning","arxiv_id":"2007.14430","date":"2020-07-28","proceeding":"NeurIPS 2020 12","authors":["Nino Vieillard","Olivier Pietquin","Matthieu Geist"],"abstract":"Bootstrapping is a core mechanism in Reinforcement Learning (RL). Most algorithms, based on temporal differences, replace the true value of a transiting state by their current estimate of this value. Yet, another estimate could be leveraged to bootstrap RL: the current policy. Our core contribution stands in a very simple idea: adding the scaled log-policy to the immediate reward. We show that slightly modifying Deep Q-Network (DQN) in that way provides an agent that is competitive with distributional methods on Atari games, without making use of distributional RL, n-step returns or prioritized replay. To demonstrate the versatility of this idea, we also use it together with an Implicit Quantile Network (IQN). The resulting agent outperforms Rainbow on Atari, installing a new State of the Art with very little modifications to the original algorithm. To add to this empirical study, we provide strong theoretical insights on what happens under the hood -- implicit Kullback-Leibler regularization and increase of the action-gap.","url_abs":"https://arxiv.org/abs/2007.14430v3","url_pdf":"https://arxiv.org/pdf/2007.14430v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"munchausen-reinforcement-learning","repo_url":"https://github.com/google-research/google-research/tree/master/munchausen_rl","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null},{"paper_slug":"munchausen-reinforcement-learning","repo_url":"https://github.com/BY571/Munchausen-RL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"munchausen-reinforcement-learning","repo_url":"https://github.com/floringogianu/atari-agents","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"munchausen-reinforcement-learning","repo_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"munchausen-reinforcement-learning","repo_url":"https://github.com/opendilab/DI-engine","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"munchausen-reinforcement-learning","repo_url":"https://github.com/deepmind/acme/tree/master/acme/agents/jax/dqn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"n-step-returns","method_name":"N-step Returns"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/atari-games-on-atari-57","task":"Atari Games","dataset":"Atari-57","model":"M-IQN","rank_in_archive_order":10,"of":11,"metrics":{"Mean Human Normalized Score":"504%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2007.14430","atlas_url":"https://app.syntology.ai/?focus=2007.14430","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2007.14430"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/floringogianu/atari-agents","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/acme/tree/master/acme/agents/jax/dqn","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/google-research/tree/master/munchausen_rl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opendilab/DI-engine","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BY571/Munchausen-RL","reach":{"status":"ok"}}],"summary":{"ran":11,"ran_violates":1,"unverified":10},"by_repo_kind":{"listed":{"samples":22,"ran":12,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5755feadde38c8e8","entry":"CateoricalPolicy","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5755feadde38c8e8"}},{"code_sha256_prefix":"81e3bb6b8f3a8963","entry":"DQNBase","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81e3bb6b8f3a8963"}},{"code_sha256_prefix":"3dafc0e0e5fc8869","entry":"LazyMemory","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3dafc0e0e5fc8869"}},{"code_sha256_prefix":"95cfd13ecea4b3d9","entry":"LazyMultiStepMemory","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95cfd13ecea4b3d9"}},{"code_sha256_prefix":"b0bd64a5291ade62","entry":"MPOParams","repo":"deepmind/acme","repo_kind":"listed","path":"acme/jax/losses/mpo.py","file_url":"https://github.com/deepmind/acme/blob/HEAD/acme/jax/losses/mpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0bd64a5291ade62"}},{"code_sha256_prefix":"3c56fd89d0f7a624","entry":"MPOStats","repo":"deepmind/acme","repo_kind":"listed","path":"acme/jax/losses/mpo.py","file_url":"https://github.com/deepmind/acme/blob/HEAD/acme/jax/losses/mpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c56fd89d0f7a624"}},{"code_sha256_prefix":"a28f79e622630932","entry":"MultiStepBuff","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a28f79e622630932"}},{"code_sha256_prefix":"acef77cb6c31e18f","entry":"MunchausenACAgent","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acef77cb6c31e18f"}},{"code_sha256_prefix":"c0deed6c236f0456","entry":"QNetwork","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0deed6c236f0456"}},{"code_sha256_prefix":"4307d53aa4781ea6","entry":"RunningMeanStats","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4307d53aa4781ea6"}},{"code_sha256_prefix":"e4242dcaea044cb7","entry":"TwinnedQNetwork","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4242dcaea044cb7"}},{"code_sha256_prefix":"26aa15e8de106e8a","entry":"compute_nonparametric_kl_from_normalized_weights","repo":"deepmind/acme","repo_kind":"listed","path":"acme/jax/losses/mpo.py","file_url":"https://github.com/deepmind/acme/blob/HEAD/acme/jax/losses/mpo.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"26aa15e8de106e8a"}},{"code_sha256_prefix":"636a9ac9309ad92c","entry":"BaseAgent","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"636a9ac9309ad92c"}},{"code_sha256_prefix":"c8be42d3d03f50ae","entry":"BaseNetwork","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8be42d3d03f50ae"}},{"code_sha256_prefix":"cd0a7dde773aaf77","entry":"LazyPrioritizedMultiStepMemory","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd0a7dde773aaf77"}},{"code_sha256_prefix":"42136719f9861313","entry":"MPO","repo":"deepmind/acme","repo_kind":"listed","path":"acme/jax/losses/mpo.py","file_url":"https://github.com/deepmind/acme/blob/HEAD/acme/jax/losses/mpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42136719f9861313"}},{"code_sha256_prefix":"6e0977288094358f","entry":"compute_cross_entropy_loss","repo":"deepmind/acme","repo_kind":"listed","path":"acme/jax/losses/mpo.py","file_url":"https://github.com/deepmind/acme/blob/HEAD/acme/jax/losses/mpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e0977288094358f"}},{"code_sha256_prefix":"fb4ceee742119787","entry":"compute_parametric_kl_penalty_and_dual_loss","repo":"deepmind/acme","repo_kind":"listed","path":"acme/jax/losses/mpo.py","file_url":"https://github.com/deepmind/acme/blob/HEAD/acme/jax/losses/mpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb4ceee742119787"}},{"code_sha256_prefix":"7fa0e09c9f031f21","entry":"compute_weights_and_temperature_loss","repo":"deepmind/acme","repo_kind":"listed","path":"acme/jax/losses/mpo.py","file_url":"https://github.com/deepmind/acme/blob/HEAD/acme/jax/losses/mpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7fa0e09c9f031f21"}},{"code_sha256_prefix":"56dd8863f4522252","entry":"disable_gradients","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56dd8863f4522252"}},{"code_sha256_prefix":"2b7b028ed940553c","entry":"initialize_weights_he","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b7b028ed940553c"}},{"code_sha256_prefix":"33523284389291bf","entry":"update_params","repo":"lingweizhu/Pytorch-MunchausenActorCritic","repo_kind":"listed","path":"Munchausen_actorcritic_discrete/agent/munchausen_ac.py","file_url":"https://github.com/lingweizhu/Pytorch-MunchausenActorCritic/blob/HEAD/Munchausen_actorcritic_discrete/agent/munchausen_ac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33523284389291bf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}