{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/instance-based-generalization-in","title":"Instance based Generalization in Reinforcement Learning","arxiv_id":"2011.01089","date":"2020-11-02","proceeding":null,"authors":["Martin Bertran","Natalia Martinez","Mariano Phielipp","Guillermo Sapiro"],"abstract":"Agents trained via deep reinforcement learning (RL) routinely fail to generalize to unseen environments, even when these share the same underlying dynamics as the training levels. Understanding the generalization properties of RL is one of the challenges of modern machine learning. Towards this goal, we analyze policy learning in the context of Partially Observable Markov Decision Processes (POMDPs) and formalize the dynamics of training levels as instances. We prove that, independently of the exploration strategy, reusing instances introduces significant changes on the effective Markov dynamics the agent observes during training. Maximizing expected rewards impacts the learned belief state of the agent by inducing undesired instance specific speedrunning policies instead of generalizeable ones, which are suboptimal on the training set. We provide generalization bounds to the value gap in train and test environments based on the number of training instances, and use insights based on these to improve performance on unseen levels. We propose training a shared belief representation over an ensemble of specialized policies, from which we compute a consensus policy that is used for data collection, disallowing instance specific exploitation. We experimentally validate our theory, observations, and the proposed computational solution over the CoinRun benchmark.","url_abs":"https://arxiv.org/abs/2011.01089v1","url_pdf":"https://arxiv.org/pdf/2011.01089v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"instance-based-generalization-in","repo_url":"https://github.com/MartinBertran/InstanceAgnosticPolicyEnsembles","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"generalization-bounds","task_name":"Generalization Bounds"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2011.01089","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2011.01089"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/MartinBertran/InstanceAgnosticPolicyEnsembles","reach":null}],"summary":{"ran":4,"ran_violates":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"5cf1dfb12135d362","entry":"ImpalaCNN","repo":"MartinBertran/InstanceAgnosticPolicyEnsembles","repo_kind":"official","path":"iape/iape/backend/core.py","file_url":"https://github.com/MartinBertran/InstanceAgnosticPolicyEnsembles/blob/HEAD/iape/iape/backend/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5cf1dfb12135d362"}},{"code_sha256_prefix":"62359dcd1406afdf","entry":"MetaBlock","repo":"MartinBertran/InstanceAgnosticPolicyEnsembles","repo_kind":"official","path":"iape/iape/backend/core.py","file_url":"https://github.com/MartinBertran/InstanceAgnosticPolicyEnsembles/blob/HEAD/iape/iape/backend/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"62359dcd1406afdf"}},{"code_sha256_prefix":"d2087f8eab5cbb06","entry":"RNNWDAbridgedModel","repo":"MartinBertran/InstanceAgnosticPolicyEnsembles","repo_kind":"official","path":"iape/iape/backend/core.py","file_url":"https://github.com/MartinBertran/InstanceAgnosticPolicyEnsembles/blob/HEAD/iape/iape/backend/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d2087f8eab5cbb06"}},{"code_sha256_prefix":"3d5dd3f37271653c","entry":"ResidualBlock","repo":"MartinBertran/InstanceAgnosticPolicyEnsembles","repo_kind":"official","path":"iape/iape/backend/core.py","file_url":"https://github.com/MartinBertran/InstanceAgnosticPolicyEnsembles/blob/HEAD/iape/iape/backend/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3d5dd3f37271653c"}},{"code_sha256_prefix":"c2f750d2cc520160","entry":"conv_output_shape","repo":"MartinBertran/InstanceAgnosticPolicyEnsembles","repo_kind":"official","path":"iape/iape/backend/core.py","file_url":"https://github.com/MartinBertran/InstanceAgnosticPolicyEnsembles/blob/HEAD/iape/iape/backend/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c2f750d2cc520160"}},{"code_sha256_prefix":"b2e23f1141df862a","entry":"max2d_output_shape","repo":"MartinBertran/InstanceAgnosticPolicyEnsembles","repo_kind":"official","path":"iape/iape/backend/core.py","file_url":"https://github.com/MartinBertran/InstanceAgnosticPolicyEnsembles/blob/HEAD/iape/iape/backend/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b2e23f1141df862a"}},{"code_sha256_prefix":"798b8adfc3fb11f0","entry":"RNNAbridgedActorCritic","repo":"MartinBertran/InstanceAgnosticPolicyEnsembles","repo_kind":"official","path":"iape/iape/backend/core.py","file_url":"https://github.com/MartinBertran/InstanceAgnosticPolicyEnsembles/blob/HEAD/iape/iape/backend/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"798b8adfc3fb11f0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}