{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-what-to-do-by-simulating-the-past-1","title":"Learning What To Do by Simulating the Past","arxiv_id":"2104.03946","date":"2021-04-08","proceeding":"ICLR 2021 1","authors":["David Lindner","Rohin Shah","Pieter Abbeel","Anca Dragan"],"abstract":"Since reward functions are hard to specify, recent work has focused on learning policies from human feedback. However, such approaches are impeded by the expense of acquiring such feedback. Recent work proposed that agents have access to a source of information that is effectively free: in any environment that humans have acted in, the state will already be optimized for human preferences, and thus an agent can extract information about what humans want from the state. Such learning is possible in principle, but requires simulating all possible past trajectories that could have led to the observed state. This is feasible in gridworlds, but how do we scale it to complex tasks? In this work, we show that by combining a learned feature encoder with learned inverse models, we can enable agents to simulate human actions backwards in time to infer what they must have done. The resulting algorithm is able to reproduce a specific skill in MuJoCo environments given a single state sampled from the optimal policy for that skill.","url_abs":"https://arxiv.org/abs/2104.03946v2","url_pdf":"https://arxiv.org/pdf/2104.03946v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-what-to-do-by-simulating-the-past-1","repo_url":"https://github.com/HumanCompatibleAI/deep-rlsp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"mujoco","task_name":"MuJoCo"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.03946","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.03946"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/HumanCompatibleAI/deep-rlsp","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"official":{"samples":9,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7b90883f18025f04","entry":"compute_d_last_step","repo":"HumanCompatibleAI/deep-rlsp","repo_kind":"official","path":"src/deep_rlsp/rlsp.py","file_url":"https://github.com/HumanCompatibleAI/deep-rlsp/blob/HEAD/src/deep_rlsp/rlsp.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b90883f18025f04"}},{"code_sha256_prefix":"6300d079d0db6635","entry":"baseline_state_distributions","repo":"HumanCompatibleAI/deep-rlsp","repo_kind":"official","path":"src/deep_rlsp/relative_reachability.py","file_url":"https://github.com/HumanCompatibleAI/deep-rlsp/blob/HEAD/src/deep_rlsp/relative_reachability.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6300d079d0db6635"}},{"code_sha256_prefix":"ab56e0052ce4f077","entry":"compute_feature_expectations","repo":"HumanCompatibleAI/deep-rlsp","repo_kind":"official","path":"src/deep_rlsp/rlsp.py","file_url":"https://github.com/HumanCompatibleAI/deep-rlsp/blob/HEAD/src/deep_rlsp/rlsp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab56e0052ce4f077"}},{"code_sha256_prefix":"69be236c06fdf0cd","entry":"compute_g","repo":"HumanCompatibleAI/deep-rlsp","repo_kind":"official","path":"src/deep_rlsp/rlsp.py","file_url":"https://github.com/HumanCompatibleAI/deep-rlsp/blob/HEAD/src/deep_rlsp/rlsp.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69be236c06fdf0cd"}},{"code_sha256_prefix":"d825d0c7b407d326","entry":"encode_transition_data","repo":"HumanCompatibleAI/deep-rlsp","repo_kind":"official","path":"src/deep_rlsp/model/dynamics_mdn.py","file_url":"https://github.com/HumanCompatibleAI/deep-rlsp/blob/HEAD/src/deep_rlsp/model/dynamics_mdn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d825d0c7b407d326"}},{"code_sha256_prefix":"60d43411850523aa","entry":"extract_play_data","repo":"HumanCompatibleAI/deep-rlsp","repo_kind":"official","path":"src/deep_rlsp/model/dynamics_mdn.py","file_url":"https://github.com/HumanCompatibleAI/deep-rlsp/blob/HEAD/src/deep_rlsp/model/dynamics_mdn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60d43411850523aa"}},{"code_sha256_prefix":"e233f51ce4afed18","entry":"get_coverage","repo":"HumanCompatibleAI/deep-rlsp","repo_kind":"official","path":"src/deep_rlsp/relative_reachability.py","file_url":"https://github.com/HumanCompatibleAI/deep-rlsp/blob/HEAD/src/deep_rlsp/relative_reachability.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e233f51ce4afed18"}},{"code_sha256_prefix":"609b7fc9cbb10c33","entry":"get_samples_from_trajectories","repo":"HumanCompatibleAI/deep-rlsp","repo_kind":"official","path":"src/deep_rlsp/policy_discriminator.py","file_url":"https://github.com/HumanCompatibleAI/deep-rlsp/blob/HEAD/src/deep_rlsp/policy_discriminator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"609b7fc9cbb10c33"}},{"code_sha256_prefix":"237bd2a08627e422","entry":"relative_reachability_penalty","repo":"HumanCompatibleAI/deep-rlsp","repo_kind":"official","path":"src/deep_rlsp/relative_reachability.py","file_url":"https://github.com/HumanCompatibleAI/deep-rlsp/blob/HEAD/src/deep_rlsp/relative_reachability.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"237bd2a08627e422"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}