{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kernel-metric-learning-for-in-sample-off","title":"Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies","arxiv_id":"2405.18792","date":"2024-05-29","proceeding":null,"authors":["Haanvid Lee","Tri Wahyu Guntara","Jongmin Lee","Yung-Kyun Noh","Kee-Eung Kim"],"abstract":"We consider off-policy evaluation (OPE) of deterministic target policies for reinforcement learning (RL) in environments with continuous action spaces. While it is common to use importance sampling for OPE, it suffers from high variance when the behavior policy deviates significantly from the target policy. In order to address this issue, some recent works on OPE proposed in-sample learning with importance resampling. Yet, these approaches are not applicable to deterministic target policies for continuous action spaces. To address this limitation, we propose to relax the deterministic target policy using a kernel and learn the kernel metrics that minimize the overall mean squared error of the estimated temporal difference update vector of an action value function, where the action value function is used for policy evaluation. We derive the bias and variance of the estimation error due to this relaxation and provide analytic solutions for the optimal kernel metric. In empirical studies using various test domains, we show that the OPE with in-sample learning using the kernel with optimized metric achieves significantly improved accuracy than other baselines.","url_abs":"https://arxiv.org/abs/2405.18792v1","url_pdf":"https://arxiv.org/pdf/2405.18792v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kernel-metric-learning-for-in-sample-off","repo_url":"https://github.com/haanvid/kmifqe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"metric-learning","task_name":"Metric Learning"},{"task_slug":"off-policy-evaluation","task_name":"Off-policy evaluation"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.18792","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.18792"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/haanvid/kmifqe","reach":null}],"summary":{"ran":2,"ran_fixture":3,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"a858007a9d9624d1","entry":"Critic","repo":"haanvid/kmifqe","repo_kind":"official","path":"KMIFQE.py","file_url":"https://github.com/haanvid/kmifqe/blob/HEAD/KMIFQE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a858007a9d9624d1"}},{"code_sha256_prefix":"36bb0207b41230a8","entry":"KMIFQE","repo":"haanvid/kmifqe","repo_kind":"official","path":"KMIFQE.py","file_url":"https://github.com/haanvid/kmifqe/blob/HEAD/KMIFQE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"36bb0207b41230a8"}},{"code_sha256_prefix":"375fe0e2f099808a","entry":"gaussian_kernel","repo":"haanvid/kmifqe","repo_kind":"official","path":"KMIFQE.py","file_url":"https://github.com/haanvid/kmifqe/blob/HEAD/KMIFQE.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"375fe0e2f099808a"}},{"code_sha256_prefix":"2b23095354e40d12","entry":"gaussian_kernel_dim_wise","repo":"haanvid/kmifqe","repo_kind":"official","path":"KMIFQE.py","file_url":"https://github.com/haanvid/kmifqe/blob/HEAD/KMIFQE.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b23095354e40d12"}},{"code_sha256_prefix":"1426432b272b7c1e","entry":"sample_action","repo":"haanvid/kmifqe","repo_kind":"official","path":"save_replay_buffer.py","file_url":"https://github.com/haanvid/kmifqe/blob/HEAD/save_replay_buffer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1426432b272b7c1e"}},{"code_sha256_prefix":"3b81875ee24223a6","entry":"boolean","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"3b81875ee24223a6"}},{"code_sha256_prefix":"7577a78af6c2977c","entry":"get_critic_grad_param","repo":"haanvid/kmifqe","repo_kind":"official","path":"KMIFQE.py","file_url":"https://github.com/haanvid/kmifqe/blob/HEAD/KMIFQE.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7577a78af6c2977c"}},{"code_sha256_prefix":"a3cf70458ecdb484","entry":"get_critic_hess_action","repo":"haanvid/kmifqe","repo_kind":"official","path":"KMIFQE.py","file_url":"https://github.com/haanvid/kmifqe/blob/HEAD/KMIFQE.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3cf70458ecdb484"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}