{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/welfare-and-fairness-in-multi-objective","title":"Welfare and Fairness in Multi-objective Reinforcement Learning","arxiv_id":"2212.01382","date":"2022-11-30","proceeding":null,"authors":["Zimeng Fan","Nianli Peng","Muhang Tian","Brandon Fain"],"abstract":"We study fair multi-objective reinforcement learning in which an agent must learn a policy that simultaneously achieves high reward on multiple dimensions of a vector-valued reward. Motivated by the fair resource allocation literature, we model this as an expected welfare maximization problem, for some nonlinear fair welfare function of the vector of long-term cumulative rewards. One canonical example of such a function is the Nash Social Welfare, or geometric mean, the log transform of which is also known as the Proportional Fairness objective. We show that even approximately optimal optimization of the expected Nash Social Welfare is computationally intractable even in the tabular case. Nevertheless, we provide a novel adaptation of Q-learning that combines nonlinear scalarized learning updates and non-stationary action selection to learn effective policies for optimizing nonlinear welfare functions. We show that our algorithm is provably convergent, and we demonstrate experimentally that our approach outperforms techniques based on linear scalarization, mixtures of optimal linear scalarizations, or stationary action selection for the Nash Social Welfare Objective.","url_abs":"https://arxiv.org/abs/2212.01382v5","url_pdf":"https://arxiv.org/pdf/2212.01382v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"welfare-and-fairness-in-multi-objective","repo_url":"https://github.com/MuhangTian/Fair-MORL-AAMAS","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":"multi-objective-reinforcement-learning","task_name":"Multi-Objective Reinforcement Learning"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"q-learning","method_name":"Q-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.01382","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.01382"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MuhangTian/Fair-MORL-AAMAS","reach":null}],"summary":{"ran_honours":2,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"d268e854fef1bd88","entry":"nsw","repo":"MuhangTian/Fair-MORL-AAMAS","repo_kind":"official","path":"nsw_ql.py","file_url":"https://github.com/MuhangTian/Fair-MORL-AAMAS/blob/HEAD/nsw_ql.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d268e854fef1bd88"}},{"code_sha256_prefix":"4d02a02347758c8c","entry":"nsw","repo":"MuhangTian/Fair-MORL-AAMAS","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/MuhangTian/Fair-MORL-AAMAS/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d02a02347758c8c"}},{"code_sha256_prefix":"f340121792a1ab4d","entry":"argmax_nsw","repo":"MuhangTian/Fair-MORL-AAMAS","repo_kind":"official","path":"nsw_ql.py","file_url":"https://github.com/MuhangTian/Fair-MORL-AAMAS/blob/HEAD/nsw_ql.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f340121792a1ab4d"}},{"code_sha256_prefix":"9777be0c77053aaf","entry":"argmax_nsw","repo":"MuhangTian/Fair-MORL-AAMAS","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/MuhangTian/Fair-MORL-AAMAS/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9777be0c77053aaf"}},{"code_sha256_prefix":"7a801faa14476e5c","entry":"eval_ql","repo":"MuhangTian/Fair-MORL-AAMAS","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/MuhangTian/Fair-MORL-AAMAS/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a801faa14476e5c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}