{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ag2manip-learning-novel-manipulation-skills","title":"Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations","arxiv_id":"2404.17521","date":"2024-04-26","proceeding":null,"authors":["Puhao Li","Tengyu Liu","Yuyang Li","Muzhi Han","Haoran Geng","Shu Wang","Yixin Zhu","Song-Chun Zhu","Siyuan Huang"],"abstract":"Autonomous robotic systems capable of learning novel manipulation tasks are poised to transform industries from manufacturing to service automation. However, modern methods (e.g., VIP and R3M) still face significant hurdles, notably the domain gap among robotic embodiments and the sparsity of successful task executions within specific action spaces, resulting in misaligned and ambiguous task representations. We introduce Ag2Manip (Agent-Agnostic representations for Manipulation), a framework aimed at surmounting these challenges through two key innovations: a novel agent-agnostic visual representation derived from human manipulation videos, with the specifics of embodiments obscured to enhance generalizability; and an agent-agnostic action representation abstracting a robot's kinematics to a universal agent proxy, emphasizing crucial interactions between end-effector and object. Ag2Manip's empirical validation across simulated benchmarks like FrankaKitchen, ManiSkill, and PartManip shows a 325% increase in performance, achieved without domain-specific demonstrations. Ablation studies underline the essential contributions of the visual and action representations to this success. Extending our evaluations to the real world, Ag2Manip significantly improves imitation learning success rates from 50% to 77.5%, demonstrating its effectiveness and generalizability across both simulated and physical environments.","url_abs":"https://arxiv.org/abs/2404.17521v1","url_pdf":"https://arxiv.org/pdf/2404.17521v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ag2manip-learning-novel-manipulation-skills","repo_url":"https://github.com/Xiaoyao-Li/Ag2Manip","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"imitation-learning","task_name":"Imitation Learning"}],"methods":[{"method_slug":null,"method_name":null}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2404.17521","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.17521"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Xiaoyao-Li/Ag2Manip","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"9589eca5bae9de80","entry":"check","repo":"Xiaoyao-Li/Ag2Manip","repo_kind":"official","path":"algos/utils/util.py","file_url":"https://github.com/Xiaoyao-Li/Ag2Manip/blob/HEAD/algos/utils/util.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9589eca5bae9de80"}},{"code_sha256_prefix":"9de355e93051e4ad","entry":"init","repo":"Xiaoyao-Li/Ag2Manip","repo_kind":"official","path":"algos/utils/util.py","file_url":"https://github.com/Xiaoyao-Li/Ag2Manip/blob/HEAD/algos/utils/util.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9de355e93051e4ad"}},{"code_sha256_prefix":"0944240e80ec7625","entry":"get_clones","repo":"Xiaoyao-Li/Ag2Manip","repo_kind":"official","path":"algos/utils/util.py","file_url":"https://github.com/Xiaoyao-Li/Ag2Manip/blob/HEAD/algos/utils/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0944240e80ec7625"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}