{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2512-02486","title":"Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts","arxiv_id":"2512.02486","date":"2025-12-02","proceeding":"ICLR","authors":["Zhongjian Qiao","Rui Yang","Jiafei Lyu","Xiu Li","Zhongxiang Dai","Zhuoran Yang","Siyang Gao","Shuang Qiu"],"abstract":"Single-domain offline reinforcement learning (RL) often suffers from limited data coverage, while cross-domain offline RL handles this issue by leveraging additional data from other domains with dynamics shifts. However, existing studies primarily focus on train-time robustness (handling dynamics shifts from training data), neglecting the test-time robustness against dynamics perturbations when deployed in practical scenarios. In this paper, we investigate dual (both train-time and test-time) robustness against dynamics shifts in cross-domain offline RL. We first empirically show that the policy trained with cross-domain offline RL exhibits fragility under dynamics perturbations during evaluation, particularly when target domain data is limited. To address this, we introduce a novel robust cross-domain Bellman (RCB) operator, which enhances test-time robustness against dynamics perturbations while staying conservative to the out-of-distribution dynamics transitions, thus guaranteeing the train-time robustness. To further counteract potential value overestimation or underestimation caused by the RCB operator, we introduce two techniques, the dynamic value penalty and the Huber loss, into our framework, resulting in the practical \\textbf{D}ual-\\textbf{RO}bust \\textbf{C}ross-domain \\textbf{O}ffline RL (DROCO) algorithm. Extensive empirical results across various dynamics shift scenarios show that DROCO outperforms strong baselines and exhibits enhanced robustness to dynamics perturbations.","url_abs":"https://arxiv.org/abs/2512.02486","url_pdf":"https://arxiv.org/pdf/2512.02486","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2512.02486","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2512.02486"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/zq2r/DROCO","reach":null}],"summary":{"ran":8,"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":13,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"61f889f4af38b19e","entry":"DoubleQFunc","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61f889f4af38b19e"}},{"code_sha256_prefix":"565d5f3b2d92ee92","entry":"EnsembleFC","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"565d5f3b2d92ee92"}},{"code_sha256_prefix":"d947cb060e01ed70","entry":"EnsembleModel","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d947cb060e01ed70"}},{"code_sha256_prefix":"c62cb3511f9129eb","entry":"MLPNetwork","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c62cb3511f9129eb"}},{"code_sha256_prefix":"8165b4e4e90023a0","entry":"Normalizer","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8165b4e4e90023a0"}},{"code_sha256_prefix":"8a661ac1c9e4bfbf","entry":"Policy","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a661ac1c9e4bfbf"}},{"code_sha256_prefix":"b6adb9bd3187839e","entry":"TanhTransform","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6adb9bd3187839e"}},{"code_sha256_prefix":"e3a1ce468acbf463","entry":"ValueFunc","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3a1ce468acbf463"}},{"code_sha256_prefix":"82a8f299419d4d83","entry":"asymmetric_l2_loss","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"82a8f299419d4d83"}},{"code_sha256_prefix":"ad44a67c8db05805","entry":"huber_loss","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad44a67c8db05805"}},{"code_sha256_prefix":"982023d532c1dda2","entry":"DROCO","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"982023d532c1dda2"}},{"code_sha256_prefix":"0e8624fb2f901e80","entry":"EnsembleDynamicsModel","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e8624fb2f901e80"}},{"code_sha256_prefix":"4cdbd80b8e624adc","entry":"Module","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cdbd80b8e624adc"}},{"code_sha256_prefix":"8076d9d6464f513f","entry":"init_weights","repo":"zq2r/DROCO","repo_kind":"found_in_text","path":"algo/offline_offline/droco.py","file_url":"https://github.com/zq2r/DROCO/blob/HEAD/algo/offline_offline/droco.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8076d9d6464f513f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}