{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/proximal-learning-with-opponent-learning","title":"Proximal Learning With Opponent-Learning Awareness","arxiv_id":"2210.10125","date":"2022-10-18","proceeding":null,"authors":["Stephen Zhao","Chris Lu","Roger Baker Grosse","Jakob Nicolaus Foerster"],"abstract":"Learning With Opponent-Learning Awareness (LOLA) (Foerster et al. [2018a]) is a multi-agent reinforcement learning algorithm that typically learns reciprocity-based cooperation in partially competitive environments. However, LOLA often fails to learn such behaviour on more complex policy spaces parameterized by neural networks, partly because the update rule is sensitive to the policy parameterization. This problem is especially pronounced in the opponent modeling setting, where the opponent's policy is unknown and must be inferred from observations; in such settings, LOLA is ill-specified because behaviorally equivalent opponent policies can result in non-equivalent updates. To address this shortcoming, we reinterpret LOLA as approximating a proximal operator, and then derive a new algorithm, proximal LOLA (POLA), which uses the proximal formulation directly. Unlike LOLA, the POLA updates are parameterization invariant, in the sense that when the proximal objective has a unique optimum, behaviorally equivalent policies result in behaviorally equivalent updates. We then present practical approximations to the ideal POLA update, which we evaluate in several partially competitive environments with function approximation and opponent modeling. This empirically demonstrates that POLA achieves reciprocity-based cooperation more reliably than LOLA.","url_abs":"https://arxiv.org/abs/2210.10125v1","url_pdf":"https://arxiv.org/pdf/2210.10125v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"proximal-learning-with-opponent-learning","repo_url":"https://github.com/silent-zebra/pola","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multi-agent-reinforcement-learning","task_name":"Multi-agent Reinforcement Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.10125","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.10125"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/alexis-jacq/LOLA_DiCE","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Silent-Zebra/POLA","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/alshedivat/lola","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/aletcher/stable-opponent-shaping","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/silent-zebra/pola","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":5,"ran_fixture":1,"ran_draft_wrong":1,"unverified":13},"by_repo_kind":{"official":{"samples":19,"ran":7,"repositories":1},"found_in_text":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"85d7bd3c92bc8523","entry":"bin_inttensor_from_int","repo":"silent-zebra/pola","repo_kind":"official","path":"LOLA_exact.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/LOLA_exact.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85d7bd3c92bc8523"}},{"code_sha256_prefix":"d21cebee917ab27f","entry":"build_bin_matrix","repo":"silent-zebra/pola","repo_kind":"official","path":"LOLA_exact.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/LOLA_exact.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d21cebee917ab27f"}},{"code_sha256_prefix":"321a0e5deba583ee","entry":"build_p_vector","repo":"silent-zebra/pola","repo_kind":"official","path":"LOLA_exact.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/LOLA_exact.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"321a0e5deba583ee"}},{"code_sha256_prefix":"70a992b1af899b2d","entry":"kl_div_jax","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70a992b1af899b2d"}},{"code_sha256_prefix":"f4166f1f2230b2e7","entry":"magic_box","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4166f1f2230b2e7"}},{"code_sha256_prefix":"54e852578016625d","entry":"rev_kl_div_jax","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54e852578016625d"}},{"code_sha256_prefix":"61e0eb0133131d4c","entry":"reverse_cumsum","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61e0eb0133131d4c"}},{"code_sha256_prefix":"0c013af22fc97f3a","entry":"ExactLOLA","repo":"alshedivat/lola","repo_kind":"found_in_text","path":"lola/exact.py","file_url":"https://github.com/alshedivat/lola/blob/HEAD/lola/exact.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c013af22fc97f3a"}},{"code_sha256_prefix":"80d4ccc33bef2e8a","entry":"act","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80d4ccc33bef2e8a"}},{"code_sha256_prefix":"2d3d1538bb6f1ad4","entry":"act_w_iter_over_obs","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d3d1538bb6f1ad4"}},{"code_sha256_prefix":"77647113e31f6449","entry":"dice_objective","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77647113e31f6449"}},{"code_sha256_prefix":"ed1c009c2edc43b1","entry":"dice_objective_plus_value_loss","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed1c009c2edc43b1"}},{"code_sha256_prefix":"62939c089f70f346","entry":"do_env_rollout","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62939c089f70f346"}},{"code_sha256_prefix":"4d53c9d04908d6ae","entry":"env_step","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d53c9d04908d6ae"}},{"code_sha256_prefix":"9e8f3bdffe71694b","entry":"get_gae_advantages","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e8f3bdffe71694b"}},{"code_sha256_prefix":"42a79b923fa6137e","entry":"get_init_hidden_states","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42a79b923fa6137e"}},{"code_sha256_prefix":"ea546e2113020484","entry":"get_policies_for_states","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea546e2113020484"}},{"code_sha256_prefix":"ce0d8c80b05c9add","entry":"in_lookahead","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce0d8c80b05c9add"}},{"code_sha256_prefix":"2f998934ffa142d3","entry":"update_gae_with_delta_backwards","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f998934ffa142d3"}},{"code_sha256_prefix":"063b9d334804c7ce","entry":"value_loss","repo":"silent-zebra/pola","repo_kind":"official","path":"jax_files/POLA_dice_jax.py","file_url":"https://github.com/silent-zebra/pola/blob/HEAD/jax_files/POLA_dice_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"063b9d334804c7ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}