{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/predictive-inverse-dynamics-models-are","title":"Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation","arxiv_id":"2412.15109","date":"2024-12-19","proceeding":null,"authors":["Yang Tian","Sizhe Yang","Jia Zeng","Ping Wang","Dahua Lin","Hao Dong","Jiangmiao Pang"],"abstract":"Current efforts to learn scalable policies in robotic manipulation primarily fall into two categories: one focuses on \"action,\" which involves behavior cloning from extensive collections of robotic data, while the other emphasizes \"vision,\" enhancing model generalization by pre-training representations or generative models, also referred to as world models, using large-scale visual datasets. This paper presents an end-to-end paradigm that predicts actions using inverse dynamics models conditioned on the robot's forecasted visual states, named Predictive Inverse Dynamics Models (PIDM). By closing the loop between vision and action, the end-to-end PIDM can be a better scalable action learner. In practice, we use Transformers to process both visual states and actions, naming the model Seer. It is initially pre-trained on large-scale robotic datasets, such as DROID, and can be adapted to realworld scenarios with a little fine-tuning data. Thanks to large-scale, end-to-end training and the synergy between vision and action, Seer significantly outperforms previous methods across both simulation and real-world experiments. It achieves improvements of 13% on the LIBERO-LONG benchmark, 21% on CALVIN ABC-D, and 43% in real-world tasks. Notably, Seer sets a new state-of-the-art on CALVIN ABC-D benchmark, achieving an average length of 4.28, and exhibits superior generalization for novel objects, lighting conditions, and environments under high-intensity disturbances on real-world scenarios. Code and models are publicly available at https://github.com/OpenRobotLab/Seer/.","url_abs":"https://arxiv.org/abs/2412.15109v1","url_pdf":"https://arxiv.org/pdf/2412.15109v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"predictive-inverse-dynamics-models-are","repo_url":"https://github.com/openrobotlab/seer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.15109","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.15109"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openrobotlab/seer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran_violates":2,"ran_fixture":1,"ran":1,"unverified":9},"by_repo_kind":{"official":{"samples":14,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4d75570b4e91d0b4","entry":"FeedForward","repo":"openrobotlab/seer","repo_kind":"official","path":"models/perceiver_resampler.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/models/perceiver_resampler.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4d75570b4e91d0b4"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"openrobotlab/seer","repo_kind":"official","path":"models/perceiver_resampler.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/models/perceiver_resampler.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"f8dd22c2d14d4521","entry":"generate_attention_mask","repo":"openrobotlab/seer","repo_kind":"official","path":"models/seer_model.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/models/seer_model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8dd22c2d14d4521"}},{"code_sha256_prefix":"5c8ea92d43ead77d","entry":"get_1d_sincos_pos_embed_from_grid","repo":"openrobotlab/seer","repo_kind":"official","path":"models/seer_model.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/models/seer_model.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c8ea92d43ead77d"}},{"code_sha256_prefix":"77e8a3ac46f3afec","entry":"get_2d_sincos_pos_embed","repo":"openrobotlab/seer","repo_kind":"official","path":"models/vit_mae.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/models/vit_mae.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77e8a3ac46f3afec"}},{"code_sha256_prefix":"2884deadbb637c3e","entry":"get_2d_sincos_pos_embed_from_grid","repo":"openrobotlab/seer","repo_kind":"official","path":"models/seer_model.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/models/seer_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2884deadbb637c3e"}},{"code_sha256_prefix":"534bfc95bfa28a2b","entry":"get_fk_solution","repo":"openrobotlab/seer","repo_kind":"official","path":"real_preprocess/convert_public_droid_to_h5_per_step.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/real_preprocess/convert_public_droid_to_h5_per_step.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"534bfc95bfa28a2b"}},{"code_sha256_prefix":"9e1f547dfc43972f","entry":"get_tf_mat","repo":"openrobotlab/seer","repo_kind":"official","path":"real_preprocess/convert_public_droid_to_h5_per_step.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/real_preprocess/convert_public_droid_to_h5_per_step.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e1f547dfc43972f"}},{"code_sha256_prefix":"f8a523b9d25c4fd9","entry":"pose_to_6d","repo":"openrobotlab/seer","repo_kind":"official","path":"deploy.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/deploy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8a523b9d25c4fd9"}},{"code_sha256_prefix":"d33079790450b952","entry":"pose_to_6d","repo":"openrobotlab/seer","repo_kind":"official","path":"real_preprocess/convert_public_droid_to_h5_per_step.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/real_preprocess/convert_public_droid_to_h5_per_step.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d33079790450b952"}},{"code_sha256_prefix":"5bd13b542fced7f7","entry":"to_np_dtype","repo":"openrobotlab/seer","repo_kind":"official","path":"real_preprocess/convert_tfds_to_h5_per_step_oxe_franka.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/real_preprocess/convert_tfds_to_h5_per_step_oxe_franka.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5bd13b542fced7f7"}},{"code_sha256_prefix":"5e7100d817574508","entry":"transform_action_berkeley_autolab_ur5","repo":"openrobotlab/seer","repo_kind":"official","path":"real_preprocess/convert_tfds_to_h5_per_step_oxe_others.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/real_preprocess/convert_tfds_to_h5_per_step_oxe_others.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5e7100d817574508"}},{"code_sha256_prefix":"6146a3eb5d2782a3","entry":"transform_action_jaco_play","repo":"openrobotlab/seer","repo_kind":"official","path":"real_preprocess/convert_tfds_to_h5_per_step_oxe_others.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/real_preprocess/convert_tfds_to_h5_per_step_oxe_others.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6146a3eb5d2782a3"}},{"code_sha256_prefix":"9ca27d5bc536bf0b","entry":"transformation_matrix_to_trans_and_euler","repo":"openrobotlab/seer","repo_kind":"official","path":"real_preprocess/convert_tfds_to_h5_per_step_oxe_franka.py","file_url":"https://github.com/openrobotlab/seer/blob/HEAD/real_preprocess/convert_tfds_to_h5_per_step_oxe_franka.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ca27d5bc536bf0b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}