{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/for-sale-state-action-representation-learning-1","title":"For SALE: State-Action Representation Learning for Deep Reinforcement Learning","arxiv_id":"2306.02451","date":"2023-06-04","proceeding":"NeurIPS 2023 11","authors":["Scott Fujimoto","Wei-Di Chang","Edward J. Smith","Shixiang Shane Gu","Doina Precup","David Meger"],"abstract":"In the field of reinforcement learning (RL), representation learning is a proven tool for complex image-based tasks, but is often overlooked for environments with low-level states, such as physical control problems. This paper introduces SALE, a novel approach for learning embeddings that model the nuanced interaction between state and action, enabling effective representation learning from low-level states. We extensively study the design space of these embeddings and highlight important design considerations. We integrate SALE and an adaptation of checkpoints for RL into TD3 to form the TD7 algorithm, which significantly outperforms existing continuous control algorithms. On OpenAI gym benchmark tasks, TD7 has an average performance gain of 276.7% and 50.7% over TD3 at 300k and 5M time steps, respectively, and works in both the online and offline settings.","url_abs":"https://arxiv.org/abs/2306.02451v2","url_pdf":"https://arxiv.org/pdf/2306.02451v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"for-sale-state-action-representation-learning-1","repo_url":"https://github.com/sfujim/td7","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"for-sale-state-action-representation-learning-1","repo_url":"https://github.com/seungju-k1m/sac-td3-td7","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"openai-gym","task_name":"OpenAI Gym"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"clipped-double-q-learning","method_name":"Clipped Double Q-learning"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"experience-replay","method_name":"Experience Replay"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"td3","method_name":"TD3"},{"method_slug":"target-policy-smoothing","method_name":"Target Policy Smoothing"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.02451","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.02451"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/haarnoja/sac","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/aviralkumar2907/CQL","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/sfujim/TD3_BC","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/ikostrikov/implicit_q_learning","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/SamsungLabs/tqc_pytorch","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Div99/XQL","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sfujim/td7","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/seungju-k1m/sac-td3-td7","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran_honours":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"found_in_text":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"1e71094b23b75fbe","entry":"AvgL1Norm","repo":"sfujim/TD7","repo_kind":"official","path":"TD7.py","file_url":"https://github.com/sfujim/TD7/blob/HEAD/TD7.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e71094b23b75fbe"}},{"code_sha256_prefix":"f829c1e4fecf264f","entry":"LAP_huber","repo":"sfujim/TD7","repo_kind":"official","path":"TD7.py","file_url":"https://github.com/sfujim/TD7/blob/HEAD/TD7.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f829c1e4fecf264f"}},{"code_sha256_prefix":"532a6dc0a512e37e","entry":"grad_gumbel","repo":"Div99/XQL","repo_kind":"found_in_text","path":"offline/critic.py","file_url":"https://github.com/Div99/XQL/blob/HEAD/offline/critic.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"532a6dc0a512e37e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}