{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-pareto-efficient-decision-making-via","title":"Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL","arxiv_id":"2305.00567","date":"2023-04-30","proceeding":null,"authors":["Baiting Zhu","Meihua Dang","Aditya Grover"],"abstract":"The goal of multi-objective reinforcement learning (MORL) is to learn policies that simultaneously optimize multiple competing objectives. In practice, an agent's preferences over the objectives may not be known apriori, and hence, we require policies that can generalize to arbitrary preferences at test time. In this work, we propose a new data-driven setup for offline MORL, where we wish to learn a preference-agnostic policy agent using only a finite dataset of offline demonstrations of other agents and their preferences. The key contributions of this work are two-fold. First, we introduce D4MORL, (D)atasets for MORL that are specifically designed for offline settings. It contains 1.8 million annotated demonstrations obtained by rolling out reference policies that optimize for randomly sampled preferences on 6 MuJoCo environments with 2-3 objectives each. Second, we propose Pareto-Efficient Decision Agents (PEDA), a family of offline MORL algorithms that builds and extends Decision Transformers via a novel preference-and-return-conditioned policy. Empirically, we show that PEDA closely approximates the behavioral policy on the D4MORL benchmark and provides an excellent approximation of the Pareto-front with appropriate conditioning, as measured by the hypervolume and sparsity metrics.","url_abs":"https://arxiv.org/abs/2305.00567v1","url_pdf":"https://arxiv.org/pdf/2305.00567v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-pareto-efficient-decision-making-via","repo_url":"https://github.com/baitingzbt/peda","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"multi-objective-reinforcement-learning","task_name":"Multi-Objective Reinforcement Learning"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.00567","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.00567"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/baitingzbt/peda","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"911cd287b73c7f3f","entry":"dirichlet_preference","repo":"baitingzbt/PEDA","repo_kind":"official","path":"data_generation/collect_env_new.py","file_url":"https://github.com/baitingzbt/PEDA/blob/HEAD/data_generation/collect_env_new.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"911cd287b73c7f3f"}},{"code_sha256_prefix":"690450285b154e1a","entry":"generate_one_alpha","repo":"baitingzbt/PEDA","repo_kind":"official","path":"data_generation/collect_env_new.py","file_url":"https://github.com/baitingzbt/PEDA/blob/HEAD/data_generation/collect_env_new.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"690450285b154e1a"}},{"code_sha256_prefix":"ffe3eca8ab003440","entry":"pref_grid","repo":"baitingzbt/PEDA","repo_kind":"official","path":"experiment.py","file_url":"https://github.com/baitingzbt/PEDA/blob/HEAD/experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ffe3eca8ab003440"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}