{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-bellman-complete-representations-for","title":"Learning Bellman Complete Representations for Offline Policy Evaluation","arxiv_id":"2207.05837","date":"2022-07-12","proceeding":null,"authors":["Jonathan D. Chang","Kaiwen Wang","Nathan Kallus","Wen Sun"],"abstract":"We study representation learning for Offline Reinforcement Learning (RL), focusing on the important task of Offline Policy Evaluation (OPE). Recent work shows that, in contrast to supervised learning, realizability of the Q-function is not enough for learning it. Two sufficient conditions for sample-efficient OPE are Bellman completeness and coverage. Prior work often assumes that representations satisfying these conditions are given, with results being mostly theoretical in nature. In this work, we propose BCRL, which directly learns from data an approximately linear Bellman complete representation with good coverage. With this learned representation, we perform OPE using Least Square Policy Evaluation (LSPE) with linear functions in our learned representation. We present an end-to-end theoretical analysis, showing that our two-stage algorithm enjoys polynomial sample complexity provided some representation in the rich class considered is linear Bellman complete. Empirically, we extensively evaluate our algorithm on challenging, image-based continuous control tasks from the Deepmind Control Suite. We show our representation enables better OPE compared to previous representation learning methods developed for off-policy RL (e.g., CURL, SPR). BCRL achieve competitive OPE error with the state-of-the-art method Fitted Q-Evaluation (FQE), and beats FQE when evaluating beyond the initial state distribution. Our ablations show that both linear Bellman complete and coverage components of our method are crucial.","url_abs":"https://arxiv.org/abs/2207.05837v1","url_pdf":"https://arxiv.org/pdf/2207.05837v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-bellman-complete-representations-for","repo_url":"https://github.com/causalml/bcrl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.05837","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.05837"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/CausalML/bcrl","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/causalml/bcrl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5b1fa9891b2abef9","entry":"Actor","repo":"causalml/bcrl","repo_kind":"official","path":"bcrl/agent.py","file_url":"https://github.com/causalml/bcrl/blob/HEAD/bcrl/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b1fa9891b2abef9"}},{"code_sha256_prefix":"9997a720459478f5","entry":"Aggregator","repo":"causalml/bcrl","repo_kind":"official","path":"bcrl/agent.py","file_url":"https://github.com/causalml/bcrl/blob/HEAD/bcrl/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9997a720459478f5"}},{"code_sha256_prefix":"47b404080ee6c37d","entry":"Encoder","repo":"causalml/bcrl","repo_kind":"official","path":"bcrl/agent.py","file_url":"https://github.com/causalml/bcrl/blob/HEAD/bcrl/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47b404080ee6c37d"}},{"code_sha256_prefix":"378d99fe8d624204","entry":"M","repo":"causalml/bcrl","repo_kind":"official","path":"bcrl/agent.py","file_url":"https://github.com/causalml/bcrl/blob/HEAD/bcrl/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"378d99fe8d624204"}},{"code_sha256_prefix":"720a41a0b42e13ae","entry":"Phi","repo":"causalml/bcrl","repo_kind":"official","path":"bcrl/agent.py","file_url":"https://github.com/causalml/bcrl/blob/HEAD/bcrl/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"720a41a0b42e13ae"}},{"code_sha256_prefix":"1f049e49b3cc3fbd","entry":"BCRL","repo":"causalml/bcrl","repo_kind":"official","path":"bcrl/agent.py","file_url":"https://github.com/causalml/bcrl/blob/HEAD/bcrl/agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f049e49b3cc3fbd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}