{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/c-morl-multi-objective-reinforcement-learning","title":"C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Front","arxiv_id":"2410.02236","date":"2024-10-03","proceeding":null,"authors":["Ruohong Liu","Yuxin Pan","Linjie Xu","Lei Song","Jiang Bian","Pengcheng You","Yize Chen"],"abstract":"Multi-objective reinforcement learning (MORL) excels at handling rapidly changing preferences in tasks that involve multiple criteria, even for unseen preferences. However, previous dominating MORL methods typically generate a fixed policy set or preference-conditioned policy through multiple training iterations exclusively for sampled preference vectors, and cannot ensure the efficient discovery of the Pareto front. Furthermore, integrating preferences into the input of policy or value functions presents scalability challenges, in particular as the dimension of the state and preference space grow, which can complicate the learning process and hinder the algorithm's performance on more complex tasks. To address these issues, we propose a two-stage Pareto front discovery algorithm called Constrained MORL (C-MORL), which serves as a seamless bridge between constrained policy optimization and MORL. Concretely, a set of policies is trained in parallel in the initialization stage, with each optimized towards its individual preference over the multiple objectives. Then, to fill the remaining vacancies in the Pareto front, the constrained optimization steps are employed to maximize one objective while constraining the other objectives to exceed a predefined threshold. Empirically, compared to recent advancements in MORL methods, our algorithm achieves more consistent and superior performances in terms of hypervolume, expected utility, and sparsity on both discrete and continuous control tasks, especially with numerous objectives (up to nine objectives in our experiments).","url_abs":"https://arxiv.org/abs/2410.02236v2","url_pdf":"https://arxiv.org/pdf/2410.02236v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"c-morl-multi-objective-reinforcement-learning","repo_url":"https://github.com/ruohliuq/c-morl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"multi-objective-reinforcement-learning","task_name":"Multi-Objective Reinforcement Learning"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.02236","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.02236"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/RuohLiuq/C-MORL","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruohliuq/c-morl","reach":{"status":"ok"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"1a080a1023e2bc1e","entry":"check_dominated","repo":"RuohLiuq/C-MORL","repo_kind":"official","path":"morl/utils.py","file_url":"https://github.com/RuohLiuq/C-MORL/blob/HEAD/morl/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a080a1023e2bc1e"}},{"code_sha256_prefix":"35fffdf717ade198","entry":"get_ep_indices","repo":"RuohLiuq/C-MORL","repo_kind":"official","path":"morl/utils.py","file_url":"https://github.com/RuohLiuq/C-MORL/blob/HEAD/morl/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35fffdf717ade198"}},{"code_sha256_prefix":"ba811cc5409d0610","entry":"update_ep","repo":"RuohLiuq/C-MORL","repo_kind":"official","path":"morl/utils.py","file_url":"https://github.com/RuohLiuq/C-MORL/blob/HEAD/morl/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba811cc5409d0610"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}