{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-04518","title":"Learning the Value Systems of Agents with Preference-based and Inverse Reinforcement Learning","arxiv_id":"2602.04518","date":"2026-02-04","proceeding":null,"authors":["Andrés Holgado-Sánchez","Holger Billhardt","Alberto Fernández","Sascha Ossowski"],"abstract":"Agreement Technologies refer to open computer systems in which autonomous software agents interact with one another, typically on behalf of humans, in order to come to mutually acceptable agreements. With the advance of AI systems in recent years, it has become apparent that such agreements, in order to be acceptable to the involved parties, must remain aligned with ethical principles and moral values. However, this is notoriously difficult to ensure, especially as different human users (and their software agents) may hold different value systems, i.e. they may differently weigh the importance of individual moral values. Furthermore, it is often hard to specify the precise meaning of a value in a particular context in a computational manner. Methods to estimate value systems based on human-engineered specifications, e.g. based on value surveys, are limited in scale due to the need for intense human moderation. In this article, we propose a novel method to automatically \\emph{learn} value systems from observations and human demonstrations. In particular, we propose a formal model of the \\emph{value system learning} problem, its instantiation to sequential decision-making domains based on multi-objective Markov decision processes, as well as tailored preference-based and inverse reinforcement learning algorithms to infer value grounding functions and value systems. The approach is illustrated and evaluated by two simulated use cases.","url_abs":"https://arxiv.org/abs/2602.04518","url_pdf":"https://arxiv.org/pdf/2602.04518","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.04518","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.04518"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/andresh26-uam/VAE-ValueLearning","reach":null}],"summary":{"unverified":8},"by_repo_kind":{"found_in_text":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e62a74f3a6f1c907","entry":"convert","repo":"andresh26-uam/VAE-ValueLearning","repo_kind":"found_in_text","path":"ValueLearningFromPreferences/src/utils.py","file_url":"https://github.com/andresh26-uam/VAE-ValueLearning/blob/HEAD/ValueLearningFromPreferences/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e62a74f3a6f1c907"}},{"code_sha256_prefix":"1295b6cac1b37be5","entry":"deconvert","repo":"andresh26-uam/VAE-ValueLearning","repo_kind":"found_in_text","path":"ValueLearningFromPreferences/src/utils.py","file_url":"https://github.com/andresh26-uam/VAE-ValueLearning/blob/HEAD/ValueLearningFromPreferences/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1295b6cac1b37be5"}},{"code_sha256_prefix":"fdaec620a1d5ff19","entry":"one_hot_encoding","repo":"andresh26-uam/VAE-ValueLearning","repo_kind":"found_in_text","path":"ValueLearningFromPreferences/src/feature_extractors.py","file_url":"https://github.com/andresh26-uam/VAE-ValueLearning/blob/HEAD/ValueLearningFromPreferences/src/feature_extractors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fdaec620a1d5ff19"}},{"code_sha256_prefix":"42784918b3440bb1","entry":"parse_cluster_sizes","repo":"andresh26-uam/VAE-ValueLearning","repo_kind":"found_in_text","path":"ValueLearningFromPreferences/train_vsl.py","file_url":"https://github.com/andresh26-uam/VAE-ValueLearning/blob/HEAD/ValueLearningFromPreferences/train_vsl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42784918b3440bb1"}},{"code_sha256_prefix":"f6d9a362e3de2c97","entry":"parse_dtype_numpy","repo":"andresh26-uam/VAE-ValueLearning","repo_kind":"found_in_text","path":"ValueLearningFromPreferences/generate_dataset_one_shot_tasks.py","file_url":"https://github.com/andresh26-uam/VAE-ValueLearning/blob/HEAD/ValueLearningFromPreferences/generate_dataset_one_shot_tasks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6d9a362e3de2c97"}},{"code_sha256_prefix":"d589fe4afdaa0dcd","entry":"parse_dtype_torch","repo":"andresh26-uam/VAE-ValueLearning","repo_kind":"found_in_text","path":"ValueLearningFromPreferences/generate_dataset_one_shot_tasks.py","file_url":"https://github.com/andresh26-uam/VAE-ValueLearning/blob/HEAD/ValueLearningFromPreferences/generate_dataset_one_shot_tasks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d589fe4afdaa0dcd"}},{"code_sha256_prefix":"da893482560569e3","entry":"parse_enames_list","repo":"andresh26-uam/VAE-ValueLearning","repo_kind":"found_in_text","path":"ValueLearningFromPreferences/evaluate_results.py","file_url":"https://github.com/andresh26-uam/VAE-ValueLearning/blob/HEAD/ValueLearningFromPreferences/evaluate_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da893482560569e3"}},{"code_sha256_prefix":"859c345b730fbdcd","entry":"train_test_split_initial_state_distributions","repo":"andresh26-uam/VAE-ValueLearning","repo_kind":"found_in_text","path":"ValueLearningFromPreferences/src/utils.py","file_url":"https://github.com/andresh26-uam/VAE-ValueLearning/blob/HEAD/ValueLearningFromPreferences/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"859c345b730fbdcd"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}