{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/optimistic-linear-support-and-successor","title":"Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer","arxiv_id":"2206.11326","date":"2022-06-22","proceeding":null,"authors":["Lucas N. Alegre","Ana L. C. Bazzan","Bruno C. da Silva"],"abstract":"In many real-world applications, reinforcement learning (RL) agents might have to solve multiple tasks, each one typically modeled via a reward function. If reward functions are expressed linearly, and the agent has previously learned a set of policies for different tasks, successor features (SFs) can be exploited to combine such policies and identify reasonable solutions for new problems. However, the identified solutions are not guaranteed to be optimal. We introduce a novel algorithm that addresses this limitation. It allows RL agents to combine existing policies and directly identify optimal policies for arbitrary new problems, without requiring any further interactions with the environment. We first show (under mild assumptions) that the transfer learning problem tackled by SFs is equivalent to the problem of learning to optimize multiple objectives in RL. We then introduce an SF-based extension of the Optimistic Linear Support algorithm to learn a set of policies whose SFs form a convex coverage set. We prove that policies in this set can be combined via generalized policy improvement to construct optimal behaviors for any new linearly-expressible tasks, without requiring any additional training samples. We empirically show that our method outperforms state-of-the-art competing algorithms both in discrete and continuous domains under value function approximation.","url_abs":"https://arxiv.org/abs/2206.11326v1","url_pdf":"https://arxiv.org/pdf/2206.11326v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimistic-linear-support-and-successor","repo_url":"https://github.com/lucasalegre/sfols","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.11326","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.11326"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LucasAlegre/sfols","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucasalegre/sfols","reach":{"status":"ok"}}],"summary":{"ran":1,"ran_draft_wrong":1,"ran_fixture":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"74a778b4b8623cae","entry":"OLS","repo":"lucasalegre/sfols","repo_kind":"official","path":"rl/successor_features/ols.py","file_url":"https://github.com/lucasalegre/sfols/blob/HEAD/rl/successor_features/ols.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74a778b4b8623cae"}},{"code_sha256_prefix":"e28c481ccf6c0899","entry":"eval_mo","repo":"lucasalegre/sfols","repo_kind":"official","path":"rl/successor_features/ols.py","file_url":"https://github.com/lucasalegre/sfols/blob/HEAD/rl/successor_features/ols.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e28c481ccf6c0899"}},{"code_sha256_prefix":"ca4adf71550a9666","entry":"policy_evaluation_mo","repo":"lucasalegre/sfols","repo_kind":"official","path":"rl/successor_features/ols.py","file_url":"https://github.com/lucasalegre/sfols/blob/HEAD/rl/successor_features/ols.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca4adf71550a9666"}},{"code_sha256_prefix":"b239b70d0aa09e5f","entry":"random_weights","repo":"lucasalegre/sfols","repo_kind":"official","path":"rl/successor_features/ols.py","file_url":"https://github.com/lucasalegre/sfols/blob/HEAD/rl/successor_features/ols.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b239b70d0aa09e5f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}