{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adversarial-intrinsic-motivation-for","title":"Adversarial Intrinsic Motivation for Reinforcement Learning","arxiv_id":"2105.13345","date":"2021-05-27","proceeding":"NeurIPS 2021 12","authors":["Ishan Durugkar","Mauricio Tec","Scott Niekum","Peter Stone"],"abstract":"Learning with an objective to minimize the mismatch with a reference distribution has been shown to be useful for generative modeling and imitation learning. In this paper, we investigate whether one such objective, the Wasserstein-1 distance between a policy's state visitation distribution and a target distribution, can be utilized effectively for reinforcement learning (RL) tasks. Specifically, this paper focuses on goal-conditioned reinforcement learning where the idealized (unachievable) target distribution has full measure at the goal. This paper introduces a quasimetric specific to Markov Decision Processes (MDPs) and uses this quasimetric to estimate the above Wasserstein-1 distance. It further shows that the policy that minimizes this Wasserstein-1 distance is the policy that reaches the goal in as few steps as possible. Our approach, termed Adversarial Intrinsic Motivation (AIM), estimates this Wasserstein-1 distance through its dual objective and uses it to compute a supplemental reward function. Our experiments show that this reward function changes smoothly with respect to transitions in the MDP and directs the agent's exploration to find the goal efficiently. Additionally, we combine AIM with Hindsight Experience Replay (HER) and show that the resulting algorithm accelerates learning significantly on several simulated robotics tasks when compared to other rewards that encourage exploration or accelerate learning.","url_abs":"https://arxiv.org/abs/2105.13345v3","url_pdf":"https://arxiv.org/pdf/2105.13345v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adversarial-intrinsic-motivation-for","repo_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multi-goal-reinforcement-learning","task_name":"Multi-Goal Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"experience-replay","method_name":"Experience Replay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2105.13345","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.13345"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":14},"by_repo_kind":{"official":{"samples":14,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5260262d19bd2712","entry":"conv","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"stable_baselines/common/tf_layers.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/stable_baselines/common/tf_layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5260262d19bd2712"}},{"code_sha256_prefix":"7bbab22bcf0e48c2","entry":"make_output_format","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"stable_baselines/logger.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/stable_baselines/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7bbab22bcf0e48c2"}},{"code_sha256_prefix":"dcefde30166a3545","entry":"millions","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"utils/plot.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/utils/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dcefde30166a3545"}},{"code_sha256_prefix":"b1b0b13946e3393b","entry":"mlp","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"stable_baselines/common/tf_layers.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/stable_baselines/common/tf_layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1b0b13946e3393b"}},{"code_sha256_prefix":"3c1cd91be28d7d5b","entry":"moving_average","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"utils/plot.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/utils/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c1cd91be28d7d5b"}},{"code_sha256_prefix":"90935b4a6d887717","entry":"ortho_init","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"stable_baselines/common/tf_layers.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/stable_baselines/common/tf_layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90935b4a6d887717"}},{"code_sha256_prefix":"a3fd9c3a978a7624","entry":"rolling_window","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"stable_baselines/results_plotter.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/stable_baselines/results_plotter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3fd9c3a978a7624"}},{"code_sha256_prefix":"7e5d730cf74c95f1","entry":"sample_a2c_params","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"utils/hyperparams_opt.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/utils/hyperparams_opt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e5d730cf74c95f1"}},{"code_sha256_prefix":"5f55ba48a52ae1f2","entry":"sample_ppo2_params","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"utils/hyperparams_opt.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/utils/hyperparams_opt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f55ba48a52ae1f2"}},{"code_sha256_prefix":"7bebd2fb30fb4073","entry":"smooth","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"utils/plot.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/utils/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7bebd2fb30fb4073"}},{"code_sha256_prefix":"75fb406b970614d4","entry":"summary_val","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"stable_baselines/logger.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/stable_baselines/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75fb406b970614d4"}},{"code_sha256_prefix":"9cc2d7b8e6215c80","entry":"ts2xy","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"stable_baselines/results_plotter.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/stable_baselines/results_plotter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cc2d7b8e6215c80"}},{"code_sha256_prefix":"dbdf57c6ccdd4893","entry":"valid_float_value","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"stable_baselines/logger.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/stable_baselines/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbdf57c6ccdd4893"}},{"code_sha256_prefix":"0680d6189aa0a2ff","entry":"window_func","repo":"iDurugkar/adversarial-intrinsic-motivation","repo_kind":"official","path":"stable_baselines/results_plotter.py","file_url":"https://github.com/iDurugkar/adversarial-intrinsic-motivation/blob/HEAD/stable_baselines/results_plotter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0680d6189aa0a2ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}