{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gr-mg-leveraging-partially-annotated-data-via","title":"GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policy","arxiv_id":"2408.14368","date":"2024-08-26","proceeding":null,"authors":["Peiyan Li","Hongtao Wu","Yan Huang","Chilam Cheang","Liang Wang","Tao Kong"],"abstract":"The robotics community has consistently aimed to achieve generalizable robot manipulation with flexible natural language instructions. One primary challenge is that obtaining robot trajectories fully annotated with both actions and texts is time-consuming and labor-intensive. However, partially-annotated data, such as human activity videos without action labels and robot trajectories without text labels, are much easier to collect. Can we leverage these data to enhance the generalization capabilities of robots? In this paper, we propose GR-MG, a novel method which supports conditioning on a text instruction and a goal image. During training, GR-MG samples goal images from trajectories and conditions on both the text and the goal image or solely on the image when text is not available. During inference, where only the text is provided, GR-MG generates the goal image via a diffusion-based image-editing model and conditions on both the text and the generated image. This approach enables GR-MG to leverage large amounts of partially-annotated data while still using languages to flexibly specify tasks. To generate accurate goal images, we propose a novel progress-guided goal image generation model which injects task progress information into the generation process. In simulation experiments, GR-MG improves the average number of tasks completed in a row of 5 from 3.35 to 4.04. In real-robot experiments, GR-MG is able to perform 58 different tasks and improves the success rate from 68.7\\% to 78.1\\% and 44.4\\% to 60.6\\% in simple and generalization settings, respectively. It also outperforms comparing baseline methods in few-shot learning of novel skills. Video demos, code, and checkpoints are available on the project page: https://gr-mg.github.io/.","url_abs":"https://arxiv.org/abs/2408.14368v2","url_pdf":"https://arxiv.org/pdf/2408.14368v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gr-mg-leveraging-partially-annotated-data-via","repo_url":"https://github.com/bytedance/GR-MG","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"robot-manipulation","task_name":"Robot Manipulation"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/robot-manipulation-on-calvin","task":"Robot Manipulation","dataset":"CALVIN","model":"GR-MG","rank_in_archive_order":6,"of":19,"metrics":{"avg. sequence length (D to D)":"4.04"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-generalization-on-calvin","task":"Zero-shot Generalization","dataset":"CALVIN","model":"GR-MG","rank_in_archive_order":1,"of":5,"metrics":{"Avg. sequence length":"4.04"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2408.14368","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.14368"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bytedance/GR-MG","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"ran_fixture":1,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5fb220ab292f1b53","entry":"alpha2rotm","repo":"bytedance/GR-MG","repo_kind":"official","path":"evaluate/utils/utils.py","file_url":"https://github.com/bytedance/GR-MG/blob/HEAD/evaluate/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5fb220ab292f1b53"}},{"code_sha256_prefix":"73992eb6c4ba4d3a","entry":"beta2rotm","repo":"bytedance/GR-MG","repo_kind":"official","path":"evaluate/utils/utils.py","file_url":"https://github.com/bytedance/GR-MG/blob/HEAD/evaluate/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73992eb6c4ba4d3a"}},{"code_sha256_prefix":"91e0e52c98282d15","entry":"count_success","repo":"bytedance/GR-MG","repo_kind":"official","path":"evaluate/utils/utils.py","file_url":"https://github.com/bytedance/GR-MG/blob/HEAD/evaluate/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"91e0e52c98282d15"}},{"code_sha256_prefix":"5c8ea92d43ead77d","entry":"get_1d_sincos_pos_embed_from_grid","repo":"bytedance/GR-MG","repo_kind":"official","path":"policy/model/vision_transformer.py","file_url":"https://github.com/bytedance/GR-MG/blob/HEAD/policy/model/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c8ea92d43ead77d"}},{"code_sha256_prefix":"5a37690a13a6157d","entry":"get_2d_sincos_pos_embed","repo":"bytedance/GR-MG","repo_kind":"official","path":"policy/model/vision_transformer.py","file_url":"https://github.com/bytedance/GR-MG/blob/HEAD/policy/model/vision_transformer.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a37690a13a6157d"}},{"code_sha256_prefix":"31528b0c209c6069","entry":"reduce_mean","repo":"bytedance/GR-MG","repo_kind":"official","path":"goal_gen/utils/dist_train.py","file_url":"https://github.com/bytedance/GR-MG/blob/HEAD/goal_gen/utils/dist_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"31528b0c209c6069"}},{"code_sha256_prefix":"d8f7af433a3feb75","entry":"reduce_sum","repo":"bytedance/GR-MG","repo_kind":"official","path":"goal_gen/utils/dist_train.py","file_url":"https://github.com/bytedance/GR-MG/blob/HEAD/goal_gen/utils/dist_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d8f7af433a3feb75"}},{"code_sha256_prefix":"7546ec17a25b5f91","entry":"reparameterize","repo":"bytedance/GR-MG","repo_kind":"official","path":"policy/model/model.py","file_url":"https://github.com/bytedance/GR-MG/blob/HEAD/policy/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7546ec17a25b5f91"}},{"code_sha256_prefix":"2884deadbb637c3e","entry":"get_2d_sincos_pos_embed_from_grid","repo":"bytedance/GR-MG","repo_kind":"official","path":"policy/model/vision_transformer.py","file_url":"https://github.com/bytedance/GR-MG/blob/HEAD/policy/model/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2884deadbb637c3e"}},{"code_sha256_prefix":"3665fea6427d655d","entry":"load_tf_weights_in_gpt2","repo":"bytedance/GR-MG","repo_kind":"official","path":"policy/model/gpt2.py","file_url":"https://github.com/bytedance/GR-MG/blob/HEAD/policy/model/gpt2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3665fea6427d655d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}