{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-human-objectives-by-evaluating","title":"Learning Human Objectives by Evaluating Hypothetical Behavior","arxiv_id":"1912.05652","date":"2019-12-05","proceeding":"ICML 2020 1","authors":["Siddharth Reddy","Anca D. Dragan","Sergey Levine","Shane Legg","Jan Leike"],"abstract":"We seek to align agent behavior with a user's objectives in a reinforcement learning setting with unknown dynamics, an unknown reward function, and unknown unsafe states. The user knows the rewards and unsafe states, but querying the user is expensive. To address this challenge, we propose an algorithm that safely and interactively learns a model of the user's reward function. We start with a generative model of initial states and a forward dynamics model trained on off-policy data. Our method uses these models to synthesize hypothetical behaviors, asks the user to label the behaviors with rewards, and trains a neural network to predict the rewards. The key idea is to actively synthesize the hypothetical behaviors from scratch by maximizing tractable proxies for the value of information, without interacting with the environment. We call this method reward query synthesis via trajectory optimization (ReQueST). We evaluate ReQueST with simulated users on a state-based 2D navigation task and the image-based Car Racing video game. The results show that ReQueST significantly outperforms prior methods in learning reward models that transfer to new environments with different initial state distributions. Moreover, ReQueST safely trains the reward model to detect unsafe states, and corrects reward hacking before deploying the agent.","url_abs":"https://arxiv.org/abs/1912.05652v2","url_pdf":"https://arxiv.org/pdf/1912.05652v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-human-objectives-by-evaluating","repo_url":"https://github.com/rddy/ReQueST","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"car-racing","task_name":"Car Racing"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1912.05652","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1912.05652"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rddy/ReQueST","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"listed":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ac4ed795d74a6fb6","entry":"get_tf_vars_in_scope","repo":"rddy/ReQueST","repo_kind":"listed","path":"rqst/utils.py","file_url":"https://github.com/rddy/ReQueST/blob/HEAD/rqst/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac4ed795d74a6fb6"}},{"code_sha256_prefix":"abd8ae82a1619b9e","entry":"make_tf_session","repo":"rddy/ReQueST","repo_kind":"listed","path":"rqst/utils.py","file_url":"https://github.com/rddy/ReQueST/blob/HEAD/rqst/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"abd8ae82a1619b9e"}},{"code_sha256_prefix":"d3d8bc2c164d3553","entry":"synth_demo","repo":"rddy/ReQueST","repo_kind":"listed","path":"rqst/reward_models.py","file_url":"https://github.com/rddy/ReQueST/blob/HEAD/rqst/reward_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d3d8bc2c164d3553"}},{"code_sha256_prefix":"6f50cb7d69f380e1","entry":"tf_logit","repo":"rddy/ReQueST","repo_kind":"listed","path":"rqst/utils.py","file_url":"https://github.com/rddy/ReQueST/blob/HEAD/rqst/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f50cb7d69f380e1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}