{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rocket-1-master-open-world-interaction-with","title":"ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting","arxiv_id":"2410.17856","date":"2024-10-23","proceeding":"CVPR 2025 1","authors":["Shaofei Cai","ZiHao Wang","Kewei Lian","Zhancun Mu","Xiaojian Ma","Anji Liu","Yitao Liang"],"abstract":"Vision-language models (VLMs) have excelled in multimodal tasks, but adapting them to embodied decision-making in open-world environments presents challenges. One critical issue is bridging the gap between discrete entities in low-level observations and the abstract concepts required for effective planning. A common solution is building hierarchical agents, where VLMs serve as high-level reasoners that break down tasks into executable sub-tasks, typically specified using language. However, language suffers from the inability to communicate detailed spatial information. We propose visual-temporal context prompting, a novel communication protocol between VLMs and policy models. This protocol leverages object segmentation from past observations to guide policy-environment interactions. Using this approach, we train ROCKET-1, a low-level policy that predicts actions based on concatenated visual observations and segmentation masks, supported by real-time object tracking from SAM-2. Our method unlocks the potential of VLMs, enabling them to tackle complex tasks that demand spatial reasoning. Experiments in Minecraft show that our approach enables agents to achieve previously unattainable tasks, with a $\\mathbf{76}\\%$ absolute improvement in open-world interaction performance. Codes and demos are now available on the project page: https://craftjarvis.github.io/ROCKET-1.","url_abs":"https://arxiv.org/abs/2410.17856v2","url_pdf":"https://arxiv.org/pdf/2410.17856v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rocket-1-master-open-world-interaction-with","repo_url":"https://github.com/CraftJarvis/ROCKET-1","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"minecraft","task_name":"Minecraft"},{"task_slug":"object-tracking","task_name":"Object Tracking"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"spatial-reasoning","task_name":"Spatial Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.17856","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.17856"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/CraftJarvis/ROCKET-1","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"33bd29cf7e72373a","entry":"clear_memory_fn","repo":"CraftJarvis/ROCKET-1","repo_kind":"official","path":"rocket/arm/eval_rocket.py","file_url":"https://github.com/CraftJarvis/ROCKET-1/blob/HEAD/rocket/arm/eval_rocket.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"33bd29cf7e72373a"}},{"code_sha256_prefix":"7632e7788be45c55","entry":"reset_fn","repo":"CraftJarvis/ROCKET-1","repo_kind":"official","path":"rocket/arm/eval_rocket.py","file_url":"https://github.com/CraftJarvis/ROCKET-1/blob/HEAD/rocket/arm/eval_rocket.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7632e7788be45c55"}},{"code_sha256_prefix":"0f06cd171c84fee2","entry":"step_fn","repo":"CraftJarvis/ROCKET-1","repo_kind":"official","path":"rocket/arm/eval_rocket.py","file_url":"https://github.com/CraftJarvis/ROCKET-1/blob/HEAD/rocket/arm/eval_rocket.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f06cd171c84fee2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}