{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/contrastive-behavioral-similarity-embeddings-1","title":"Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning","arxiv_id":"2101.05265","date":"2021-01-13","proceeding":"ICLR 2021 1","authors":["Rishabh Agarwal","Marlos C. Machado","Pablo Samuel Castro","Marc G. Bellemare"],"abstract":"Reinforcement learning methods trained on few environments rarely learn policies that generalize to unseen environments. To improve generalization, we incorporate the inherent sequential structure in reinforcement learning into the representation learning process. This approach is orthogonal to recent approaches, which rarely exploit this structure explicitly. Specifically, we introduce a theoretically motivated policy similarity metric (PSM) for measuring behavioral similarity between states. PSM assigns high similarity to states for which the optimal policies in those states as well as in future states are similar. We also present a contrastive representation learning procedure to embed any state similarity metric, which we instantiate with PSM to obtain policy similarity embeddings (PSEs). We demonstrate that PSEs improve generalization on diverse benchmarks, including LQR with spurious correlations, a jumping task from pixels, and Distracting DM Control Suite.","url_abs":"https://arxiv.org/abs/2101.05265v2","url_pdf":"https://arxiv.org/pdf/2101.05265v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"contrastive-behavioral-similarity-embeddings-1","repo_url":"https://github.com/google-research/google-research/tree/master/pse","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"policy-similarity-metric","method_name":"Policy Similarity Metric"}],"datasets_introduced":[],"methods_introduced":[{"slug":"policy-similarity-metric","name":"Policy Similarity Metric","full_name":"Policy Similarity Metric"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2101.05265","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}