{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-11499","title":"Offline Reinforcement Learning with Generative Trajectory Policies","arxiv_id":"2510.11499","date":"2025-10-13","proceeding":"ICML","authors":["Xinsong Feng","Leshu Tang","Chenan Wang","Haipeng Chen"],"abstract":"Generative models have emerged as a powerful class of policies for offline reinforcement learning (RL) due to their ability to capture complex, multi-modal behaviors. However, existing methods face a stark trade-off: slow, iterative models like diffusion policies are computationally expensive, while fast, single-step models like consistency policies often suffer from degraded performance. In this paper, we demonstrate that it is possible to bridge this gap. The key to moving beyond the limitations of individual methods, we argue, lies in a unifying perspective that views modern generative models, including diffusion, flow matching, and consistency models, as specific instances of learning a continuous-time generative trajectory governed by an Ordinary Differential Equation (ODE). This principled foundation provides a clearer design space for generative policies in RL and allows us to propose Generative Trajectory Policies (GTPs), a new and more general policy paradigm that learns the entire solution map of the underlying ODE. To make this paradigm practical for offline RL, we further introduce two key theoretically principled adaptations. Empirical results demonstrate that GTP achieves state-of-the-art performance on D4RL benchmarks - it significantly outperforms prior generative policies, achieving perfect scores on several notoriously hard AntMaze tasks.","url_abs":"https://arxiv.org/abs/2510.11499","url_pdf":"https://arxiv.org/pdf/2510.11499","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.11499","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.11499"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/wmd3i/gtp","reach":null}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"e3eea0015cfe01c3","entry":"WeightedHuber","repo":"wmd3i/gtp","repo_kind":"found_in_text","path":"agents/diffusion.py","file_url":"https://github.com/wmd3i/gtp/blob/HEAD/agents/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3eea0015cfe01c3"}},{"code_sha256_prefix":"2e0c2c54ebebc9bc","entry":"WeightedL1","repo":"wmd3i/gtp","repo_kind":"found_in_text","path":"agents/diffusion.py","file_url":"https://github.com/wmd3i/gtp/blob/HEAD/agents/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e0c2c54ebebc9bc"}},{"code_sha256_prefix":"56917a389275eaae","entry":"WeightedL2","repo":"wmd3i/gtp","repo_kind":"found_in_text","path":"agents/diffusion.py","file_url":"https://github.com/wmd3i/gtp/blob/HEAD/agents/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56917a389275eaae"}},{"code_sha256_prefix":"7c3e00ce2a276ede","entry":"Diffusion","repo":"wmd3i/gtp","repo_kind":"found_in_text","path":"agents/diffusion.py","file_url":"https://github.com/wmd3i/gtp/blob/HEAD/agents/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c3e00ce2a276ede"}},{"code_sha256_prefix":"427953099d88e4d0","entry":"WeightedLoss","repo":"wmd3i/gtp","repo_kind":"found_in_text","path":"agents/diffusion.py","file_url":"https://github.com/wmd3i/gtp/blob/HEAD/agents/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"427953099d88e4d0"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}