{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/double-check-your-state-before-trusting-it","title":"Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination","arxiv_id":"2206.07989","date":"2022-06-16","proceeding":null,"authors":["Jiafei Lyu","Xiu Li","Zongqing Lu"],"abstract":"The learned policy of model-free offline reinforcement learning (RL) methods is often constrained to stay within the support of datasets to avoid possible dangerous out-of-distribution actions or states, making it challenging to handle out-of-support region. Model-based RL methods offer a richer dataset and benefit generalization by generating imaginary trajectories with either trained forward or reverse dynamics model. However, the imagined transitions may be inaccurate, thus downgrading the performance of the underlying offline RL method. In this paper, we propose to augment the offline dataset by using trained bidirectional dynamics models and rollout policies with double check. We introduce conservatism by trusting samples that the forward model and backward model agree on. Our method, confidence-aware bidirectional offline model-based imagination, generates reliable samples and can be combined with any model-free offline RL method. Experimental results on the D4RL benchmarks demonstrate that our method significantly boosts the performance of existing model-free offline RL algorithms and achieves competitive or better scores against baseline methods.","url_abs":"https://arxiv.org/abs/2206.07989v2","url_pdf":"https://arxiv.org/pdf/2206.07989v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"double-check-your-state-before-trusting-it","repo_url":"https://github.com/dmksjfl/CABI","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.07989","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.07989"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tianheyu927/mopo","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/apple/ml-uwac","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/ikostrikov/implicit_q_learning","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/wenzhe-li/romi","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dmksjfl/CABI","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1},"found_in_text":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"e6fc32e81dc5b012","entry":"FakeEnv","repo":"wenzhe-li/romi","repo_kind":"found_in_text","path":"mopo/mopo/models/fake_env.py","file_url":"https://github.com/wenzhe-li/romi/blob/HEAD/mopo/mopo/models/fake_env.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6fc32e81dc5b012"}},{"code_sha256_prefix":"423a675945adddb2","entry":"get_scaler","repo":"dmksjfl/CABI","repo_kind":"official","path":"offlinerl/utils/data.py","file_url":"https://github.com/dmksjfl/CABI/blob/HEAD/offlinerl/utils/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"423a675945adddb2"}},{"code_sha256_prefix":"f8b2ddca237d584a","entry":"parse_config","repo":"dmksjfl/CABI","repo_kind":"official","path":"offlinerl/utils/config.py","file_url":"https://github.com/dmksjfl/CABI/blob/HEAD/offlinerl/utils/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f8b2ddca237d584a"}},{"code_sha256_prefix":"1c3f616e7d7b57ea","entry":"test_on_real_env","repo":"dmksjfl/CABI","repo_kind":"official","path":"offlinerl/evaluation/neorl.py","file_url":"https://github.com/dmksjfl/CABI/blob/HEAD/offlinerl/evaluation/neorl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c3f616e7d7b57ea"}},{"code_sha256_prefix":"4fc31f063206b61b","entry":"test_one_trail","repo":"dmksjfl/CABI","repo_kind":"official","path":"offlinerl/evaluation/neorl.py","file_url":"https://github.com/dmksjfl/CABI/blob/HEAD/offlinerl/evaluation/neorl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fc31f063206b61b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}