{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/state-action-similarity-based-representations-1","title":"State-Action Similarity-Based Representations for Off-Policy Evaluation","arxiv_id":"2310.18409","date":"2023-10-27","proceeding":"NeurIPS 2023 11","authors":["Brahma S. Pavse","Josiah P. Hanna"],"abstract":"In reinforcement learning, off-policy evaluation (OPE) is the problem of estimating the expected return of an evaluation policy given a fixed dataset that was collected by running one or more different policies. One of the more empirically successful algorithms for OPE has been the fitted q-evaluation (FQE) algorithm that uses temporal difference updates to learn an action-value function, which is then used to estimate the expected return of the evaluation policy. Typically, the original fixed dataset is fed directly into FQE to learn the action-value function of the evaluation policy. Instead, in this paper, we seek to enhance the data-efficiency of FQE by first transforming the fixed dataset using a learned encoder, and then feeding the transformed dataset into FQE. To learn such an encoder, we introduce an OPE-tailored state-action behavioral similarity metric, and use this metric and the fixed dataset to learn an encoder that models this metric. Theoretically, we show that this metric allows us to bound the error in the resulting OPE estimate. Empirically, we show that other state-action similarity metrics lead to representations that cannot represent the action-value function of the evaluation policy, and that our state-action representation method boosts the data-efficiency of FQE and lowers OPE error relative to other OPE-based representation learning methods on challenging OPE tasks. We also empirically show that the learned representations significantly mitigate divergence of FQE under varying distribution shifts. Our code is available here: https://github.com/Badger-RL/ROPE.","url_abs":"https://arxiv.org/abs/2310.18409v1","url_pdf":"https://arxiv.org/pdf/2310.18409v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"state-action-similarity-based-representations-1","repo_url":"https://github.com/badger-rl/rope","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"off-policy-evaluation","task_name":"Off-policy evaluation"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.18409","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.18409"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Badger-RL/ROPE","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/badger-rl/rope","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_violates":1,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"98d3b81dd3754492","entry":"NeuralNetwork","repo":"badger-rl/rope","repo_kind":"official","path":"learn_phi.py","file_url":"https://github.com/badger-rl/rope/blob/HEAD/learn_phi.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98d3b81dd3754492"}},{"code_sha256_prefix":"00f3fd9d0d56c95e","entry":"OffPolicySA","repo":"badger-rl/rope","repo_kind":"official","path":"learn_phi.py","file_url":"https://github.com/badger-rl/rope/blob/HEAD/learn_phi.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00f3fd9d0d56c95e"}},{"code_sha256_prefix":"fae4366b0e03cae3","entry":"PositivityActivation","repo":"badger-rl/rope","repo_kind":"official","path":"learn_phi.py","file_url":"https://github.com/badger-rl/rope/blob/HEAD/learn_phi.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fae4366b0e03cae3"}},{"code_sha256_prefix":"1125b171f048d4c8","entry":"Symlog","repo":"badger-rl/rope","repo_kind":"official","path":"learn_phi.py","file_url":"https://github.com/badger-rl/rope/blob/HEAD/learn_phi.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1125b171f048d4c8"}},{"code_sha256_prefix":"55e3bac9d0c163ae","entry":"compute_stats","repo":"Badger-RL/ROPE","repo_kind":"official","path":"plot.py","file_url":"https://github.com/Badger-RL/ROPE/blob/HEAD/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55e3bac9d0c163ae"}},{"code_sha256_prefix":"f017532fc389cbfe","entry":"str2bool","repo":"Badger-RL/ROPE","repo_kind":"official","path":"gen_offline_dataset.py","file_url":"https://github.com/Badger-RL/ROPE/blob/HEAD/gen_offline_dataset.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f017532fc389cbfe"}},{"code_sha256_prefix":"7f639a210564127e","entry":"best_hp","repo":"Badger-RL/ROPE","repo_kind":"official","path":"plot.py","file_url":"https://github.com/Badger-RL/ROPE/blob/HEAD/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f639a210564127e"}},{"code_sha256_prefix":"c288d13b595ca6a6","entry":"load_env","repo":"Badger-RL/ROPE","repo_kind":"official","path":"utils.py","file_url":"https://github.com/Badger-RL/ROPE/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c288d13b595ca6a6"}},{"code_sha256_prefix":"26c23657c2bc3740","entry":"soft_target_update","repo":"badger-rl/rope","repo_kind":"official","path":"learn_phi.py","file_url":"https://github.com/badger-rl/rope/blob/HEAD/learn_phi.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"26c23657c2bc3740"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}