{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-24862","title":"Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets","arxiv_id":"2605.24862","date":"2026-05-24","proceeding":"ICML","authors":["Zhongjian Qiao","Jiafei Lyu","Chenjia Bai","Peisong Wang","Siyang Gao","Shuang Qiu"],"abstract":"Cross-domain offline reinforcement learning (RL) aims to learn a policy in the target domain with a limited target domain dataset and a source domain dataset that exhibits a dynamics shift. Training directly on the original source dataset typically leads to performance collapse. Recent studies perform data filtering from the perspective of dynamics alignment or value alignment to enable efficient policy transfer. However, these studies are typically validated on single-domain or single-behavior-policy source datasets. In this work, we explore a more general heterogeneous cross-domain offline RL setting, where the source datasets may be collected from multiple source domains by diverse behavior policies. We first uncover a critical yet overlooked issue in this setting: value misassignment. Empirically and theoretically, we demonstrate that value misassignment can undermine value alignment, mislead data filtering toward selecting suboptimal samples, and loosen the suboptimality gap, thereby degrading the agent's performance. To address this issue, we propose V2A, which integrates dynamics alignment, value alignment, and value assignment. V2A first employs temporally-consistent modality representation learning to extract dynamics modalities from the source dataset, followed by modality-aware advantage learning to rectify value alignment. Finally, it adopts a data filtering paradigm to selectively share source data for policy learning. Empirical results show that V2A significantly outperforms strong baseline methods under general heterogeneous cross-domain offline RL settings.","url_abs":"https://arxiv.org/abs/2605.24862","url_pdf":"https://arxiv.org/pdf/2605.24862","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.24862","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.24862"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/zq2r/V2A","reach":null}],"summary":{"ran":10,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":12,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"4adaafff14690407","entry":"ContrastiveInfo","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4adaafff14690407"}},{"code_sha256_prefix":"83e6f21e17e7bb78","entry":"DoubleQFunc","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"83e6f21e17e7bb78"}},{"code_sha256_prefix":"b0656c14e797d023","entry":"DoubleQFuncZ","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0656c14e797d023"}},{"code_sha256_prefix":"caffdc2cc04cbc6d","entry":"LayerNormEnsemble","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"caffdc2cc04cbc6d"}},{"code_sha256_prefix":"c76593b92332a06f","entry":"LinearEnsemble","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c76593b92332a06f"}},{"code_sha256_prefix":"df54328b8327c1f5","entry":"MLPNetwork","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df54328b8327c1f5"}},{"code_sha256_prefix":"17fe358051cb051d","entry":"Policy","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17fe358051cb051d"}},{"code_sha256_prefix":"349b6c796a1430be","entry":"TanhTransform","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"349b6c796a1430be"}},{"code_sha256_prefix":"fb381367489d8939","entry":"ValueFunc","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb381367489d8939"}},{"code_sha256_prefix":"0a7cfd6595f42b67","entry":"ValueFuncZ","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a7cfd6595f42b67"}},{"code_sha256_prefix":"80731ffdd4365bf4","entry":"EnsembleMLP","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80731ffdd4365bf4"}},{"code_sha256_prefix":"d630866dbd5ad988","entry":"V2A_IGDF","repo":"zq2r/V2A","repo_kind":"found_in_text","path":"algo/offline_offline/v2a_igdf.py","file_url":"https://github.com/zq2r/V2A/blob/HEAD/algo/offline_offline/v2a_igdf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d630866dbd5ad988"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}