{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-can-i-do-here-learning-new-skills-by","title":"What Can I Do Here? Learning New Skills by Imagining Visual Affordances","arxiv_id":"2106.00671","date":"2021-06-01","proceeding":null,"authors":["Alexander Khazatsky","Ashvin Nair","Daniel Jing","Sergey Levine"],"abstract":"A generalist robot equipped with learned skills must be able to perform many tasks in many different environments. However, zero-shot generalization to new settings is not always possible. When the robot encounters a new environment or object, it may need to finetune some of its previously learned skills to accommodate this change. But crucially, previously learned behaviors and models should still be suitable to accelerate this relearning. In this paper, we aim to study how generative models of possible outcomes can allow a robot to learn visual representations of affordances, so that the robot can sample potentially possible outcomes in new situations, and then further train its policy to achieve those outcomes. In effect, prior data is used to learn what kinds of outcomes may be possible, such that when the robot encounters an unfamiliar setting, it can sample potential outcomes from its model, attempt to reach them, and thereby update both its skills and its outcome model. This approach, visuomotor affordance learning (VAL), can be used to train goal-conditioned policies that operate on raw image inputs, and can rapidly learn to manipulate new objects via our proposed affordance-directed exploration scheme. We show that VAL can utilize prior data to solve real-world tasks such drawer opening, grasping, and placing objects in new scenes with only five minutes of online experience in the new scene.","url_abs":"https://arxiv.org/abs/2106.00671v2","url_pdf":"https://arxiv.org/pdf/2106.00671v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-can-i-do-here-learning-new-skills-by","repo_url":"https://github.com/anair13/bullet-manipulation-affordances","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"what-can-i-do-here-learning-new-skills-by","repo_url":"https://github.com/patrickhaoy/ptp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[],"datasets_introduced":[{"slug":"visuomotor-affordance-learning-val-robot","name":"Visuomotor affordance learning (VAL) robot interaction dataset","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.00671","atlas_url":"https://app.syntology.ai/?focus=2106.00671","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.00671"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/anair13/bullet-manipulation-affordances","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/patrickhaoy/ptp","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"unverified":3},"by_repo_kind":{"listed":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fadf5f762ff04cd6","entry":"create_stats_ordered_dict","repo":"patrickhaoy/ptp","repo_kind":"listed","path":"rlkit/core/eval_util.py","file_url":"https://github.com/patrickhaoy/ptp/blob/HEAD/rlkit/core/eval_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fadf5f762ff04cd6"}},{"code_sha256_prefix":"a5d10a0b704c8db7","entry":"get_average_returns","repo":"patrickhaoy/ptp","repo_kind":"listed","path":"rlkit/core/eval_util.py","file_url":"https://github.com/patrickhaoy/ptp/blob/HEAD/rlkit/core/eval_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5d10a0b704c8db7"}},{"code_sha256_prefix":"6dc14f1205c31904","entry":"add_prefix","repo":"patrickhaoy/ptp","repo_kind":"listed","path":"rlkit/core/logging.py","file_url":"https://github.com/patrickhaoy/ptp/blob/HEAD/rlkit/core/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6dc14f1205c31904"}},{"code_sha256_prefix":"9817c2d99db83a09","entry":"append_log","repo":"patrickhaoy/ptp","repo_kind":"listed","path":"rlkit/core/logging.py","file_url":"https://github.com/patrickhaoy/ptp/blob/HEAD/rlkit/core/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9817c2d99db83a09"}},{"code_sha256_prefix":"26a21a70206f48ab","entry":"reopen","repo":"patrickhaoy/ptp","repo_kind":"listed","path":"rlkit/core/logging.py","file_url":"https://github.com/patrickhaoy/ptp/blob/HEAD/rlkit/core/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26a21a70206f48ab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}