{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cross-trajectory-representation-learning-for","title":"Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL","arxiv_id":"2106.02193","date":"2021-06-04","proceeding":"ICLR 2022 4","authors":["Bogdan Mazoure","Ahmed M. Ahmed","Patrick MacAlpine","R Devon Hjelm","Andrey Kolobov"],"abstract":"A highly desirable property of a reinforcement learning (RL) agent -- and a major difficulty for deep RL approaches -- is the ability to generalize policies learned on a few tasks over a high-dimensional observation space to similar tasks not seen during training. Many promising approaches to this challenge consider RL as a process of training two functions simultaneously: a complex nonlinear encoder that maps high-dimensional observations to a latent representation space, and a simple linear policy over this space. We posit that a superior encoder for zero-shot generalization in RL can be trained by using solely an auxiliary SSL objective if the training process encourages the encoder to map behaviorally similar observations to similar representations, as reward-based signal can cause overfitting in the encoder (Raileanu et al., 2021). We propose Cross-Trajectory Representation Learning (CTRL), a method that runs within an RL agent and conditions its encoder to recognize behavioral similarity in observations by applying a novel SSL objective to pairs of trajectories from the agent's policies. CTRL can be viewed as having the same effect as inducing a pseudo-bisimulation metric but, crucially, avoids the use of rewards and associated overfitting risks. Our experiments ablate various components of CTRL and demonstrate that in combination with PPO it achieves better generalization performance on the challenging Procgen benchmark suite (Cobbe et al., 2020).","url_abs":"https://arxiv.org/abs/2106.02193v2","url_pdf":"https://arxiv.org/pdf/2106.02193v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cross-trajectory-representation-learning-for","repo_url":"https://github.com/bmazoure/ctrl_public","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[{"method_slug":"adagrad","method_name":"AdaGrad"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"ctrl","method_name":"CTRL"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"gradient-clipping","method_name":"Gradient Clipping"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup","method_name":"Linear Warmup"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"ppo","method_name":"PPO"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.02193","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.02193"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bmazoure/ctrl_public","reach":null}],"summary":{"ran":6,"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"32fdabb6701ee743","entry":"MLP","repo":"bmazoure/ctrl_public","repo_kind":"official","path":"models.py","file_url":"https://github.com/bmazoure/ctrl_public/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32fdabb6701ee743"}},{"code_sha256_prefix":"bb6a2e2f9d7073da","entry":"ResidualBlock","repo":"bmazoure/ctrl_public","repo_kind":"official","path":"models.py","file_url":"https://github.com/bmazoure/ctrl_public/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb6a2e2f9d7073da"}},{"code_sha256_prefix":"5e7d81a9ce871d74","entry":"default_conv_init","repo":"bmazoure/ctrl_public","repo_kind":"official","path":"models.py","file_url":"https://github.com/bmazoure/ctrl_public/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e7d81a9ce871d74"}},{"code_sha256_prefix":"58ffb3cae4f10d34","entry":"default_linear_init","repo":"bmazoure/ctrl_public","repo_kind":"official","path":"models.py","file_url":"https://github.com/bmazoure/ctrl_public/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58ffb3cae4f10d34"}},{"code_sha256_prefix":"5d8cdfb148808635","entry":"default_logits_init","repo":"bmazoure/ctrl_public","repo_kind":"official","path":"models.py","file_url":"https://github.com/bmazoure/ctrl_public/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d8cdfb148808635"}},{"code_sha256_prefix":"fe9b930c7511fff1","entry":"default_relu_init","repo":"bmazoure/ctrl_public","repo_kind":"official","path":"models.py","file_url":"https://github.com/bmazoure/ctrl_public/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe9b930c7511fff1"}},{"code_sha256_prefix":"2ed07b9462bf332f","entry":"select_action","repo":"bmazoure/ctrl_public","repo_kind":"official","path":"algo.py","file_url":"https://github.com/bmazoure/ctrl_public/blob/HEAD/algo.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ed07b9462bf332f"}},{"code_sha256_prefix":"23cdb200dcee7eeb","entry":"CTRLModel","repo":"bmazoure/ctrl_public","repo_kind":"official","path":"models.py","file_url":"https://github.com/bmazoure/ctrl_public/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23cdb200dcee7eeb"}},{"code_sha256_prefix":"9b96a172a1cbf7d9","entry":"Impala","repo":"bmazoure/ctrl_public","repo_kind":"official","path":"models.py","file_url":"https://github.com/bmazoure/ctrl_public/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b96a172a1cbf7d9"}},{"code_sha256_prefix":"f8b1ad84240423bd","entry":"compute_distance","repo":"bmazoure/ctrl_public","repo_kind":"official","path":"algo.py","file_url":"https://github.com/bmazoure/ctrl_public/blob/HEAD/algo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8b1ad84240423bd"}},{"code_sha256_prefix":"93ff26fbeba24772","entry":"safe_mean","repo":"bmazoure/ctrl_public","repo_kind":"official","path":"train_ppo.py","file_url":"https://github.com/bmazoure/ctrl_public/blob/HEAD/train_ppo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93ff26fbeba24772"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}