{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/instruction-driven-history-aware-policies-for","title":"Instruction-driven history-aware policies for robotic manipulations","arxiv_id":"2209.04899","date":"2022-09-11","proceeding":null,"authors":["Pierre-Louis Guhur","ShiZhe Chen","Ricardo Garcia","Makarand Tapaswi","Ivan Laptev","Cordelia Schmid"],"abstract":"In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. Yet, robotic manipulation is extremely challenging as it requires fine-grained motor control, long-term memory as well as generalization to previously unseen tasks and environments. To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions. Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views. We evaluate our method on the challenging RLBench benchmark and on a real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state of the art. We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations.","url_abs":"https://arxiv.org/abs/2209.04899v3","url_pdf":"https://arxiv.org/pdf/2209.04899v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"instruction-driven-history-aware-policies-for","repo_url":"https://github.com/vlc-robot/polarnet","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"instruction-driven-history-aware-policies-for","repo_url":"https://github.com/guhur/hiveformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"robot-manipulation","task_name":"Robot Manipulation"},{"task_slug":"robot-manipulation-generalization","task_name":"Robot Manipulation Generalization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/robot-manipulation-on-rlbench","task":"Robot Manipulation","dataset":"RLBench","model":"Hiveformer","rank_in_archive_order":13,"of":18,"metrics":{"Succ. Rate (10 tasks, 100 demos/task)":"83.3","Succ. Rate (18 tasks, 100 demo/task)":"45.3"},"uses_additional_data":false},{"leaderboard":"/sota/robot-manipulation-generalization-on-gembench","task":"Robot Manipulation Generalization","dataset":"GEMBench","model":"Hiveformer","rank_in_archive_order":6,"of":6,"metrics":{"Average Success Rate":"30.4","Average Success Rate (L1)":"60.3±1.5","Average Success Rate (L2)":"26.1±1.4","Average Success Rate (L3)":"35.1±1.7","Average Success Rate (L4)":"0.0±0.0"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2209.04899","atlas_url":"https://app.syntology.ai/?focus=2209.04899","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.04899"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/guhur/hiveformer","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vlc-robot/polarnet","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":6},"by_repo_kind":{"listed":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7d979f85e8121f26","entry":"generate_mask_obs","repo":"guhur/hiveformer","repo_kind":"listed","path":"network.py","file_url":"https://github.com/guhur/hiveformer/blob/HEAD/network.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d979f85e8121f26"}},{"code_sha256_prefix":"e08b3437c448bb17","entry":"get_causal_mask_by_block","repo":"guhur/hiveformer","repo_kind":"listed","path":"network.py","file_url":"https://github.com/guhur/hiveformer/blob/HEAD/network.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e08b3437c448bb17"}},{"code_sha256_prefix":"bad211cd427d59bf","entry":"norm_tensor","repo":"guhur/hiveformer","repo_kind":"listed","path":"network.py","file_url":"https://github.com/guhur/hiveformer/blob/HEAD/network.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bad211cd427d59bf"}},{"code_sha256_prefix":"082118868638dd54","entry":"obs_to_attn","repo":"guhur/hiveformer","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/guhur/hiveformer/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"082118868638dd54"}},{"code_sha256_prefix":"e03456c6ae93a8de","entry":"parse_int","repo":"guhur/hiveformer","repo_kind":"listed","path":"preprocess_instructions.py","file_url":"https://github.com/guhur/hiveformer/blob/HEAD/preprocess_instructions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e03456c6ae93a8de"}},{"code_sha256_prefix":"272295b2b14c060e","entry":"data_transform","repo":"guhur/hiveformer","repo_kind":"listed","path":"dataset.py","file_url":"https://github.com/guhur/hiveformer/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"272295b2b14c060e"}},{"code_sha256_prefix":"33a28f41002690b5","entry":"get_max_episode_length","repo":"guhur/hiveformer","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/guhur/hiveformer/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33a28f41002690b5"}},{"code_sha256_prefix":"1eb3169facaa8376","entry":"load_model","repo":"guhur/hiveformer","repo_kind":"listed","path":"preprocess_instructions.py","file_url":"https://github.com/guhur/hiveformer/blob/HEAD/preprocess_instructions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1eb3169facaa8376"}},{"code_sha256_prefix":"1441fd48c61694c2","entry":"load_tokenizer","repo":"guhur/hiveformer","repo_kind":"listed","path":"preprocess_instructions.py","file_url":"https://github.com/guhur/hiveformer/blob/HEAD/preprocess_instructions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1441fd48c61694c2"}},{"code_sha256_prefix":"17bfe81c047d25b6","entry":"loader","repo":"guhur/hiveformer","repo_kind":"listed","path":"dataset.py","file_url":"https://github.com/guhur/hiveformer/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17bfe81c047d25b6"}},{"code_sha256_prefix":"4d877f2e65b9850e","entry":"task_file_to_task_class","repo":"guhur/hiveformer","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/guhur/hiveformer/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d877f2e65b9850e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}