{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodal-diffusion-transformer-learning","title":"Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals","arxiv_id":"2407.05996","date":"2024-07-08","proceeding":null,"authors":["Moritz Reuss","Ömer Erdinç Yağmurlu","Fabian Wenzel","Rudolf Lioutikov"],"abstract":"This work introduces the Multimodal Diffusion Transformer (MDT), a novel diffusion policy framework, that excels at learning versatile behavior from multimodal goal specifications with few language annotations. MDT leverages a diffusion-based multimodal transformer backbone and two self-supervised auxiliary objectives to master long-horizon manipulation tasks based on multimodal goals. The vast majority of imitation learning methods only learn from individual goal modalities, e.g. either language or goal images. However, existing large-scale imitation learning datasets are only partially labeled with language annotations, which prohibits current methods from learning language conditioned behavior from these datasets. MDT addresses this challenge by introducing a latent goal-conditioned state representation that is simultaneously trained on multimodal goal instructions. This state representation aligns image and language based goal embeddings and encodes sufficient information to predict future states. The representation is trained via two self-supervised auxiliary objectives, enhancing the performance of the presented transformer backbone. MDT shows exceptional performance on 164 tasks provided by the challenging CALVIN and LIBERO benchmarks, including a LIBERO version that contains less than $2\\%$ language annotations. Furthermore, MDT establishes a new record on the CALVIN manipulation challenge, demonstrating an absolute performance improvement of $15\\%$ over prior state-of-the-art methods that require large-scale pretraining and contain $10\\times$ more learnable parameters. MDT shows its ability to solve long-horizon manipulation from sparsely annotated data in both simulated and real-world environments. Demonstrations and Code are available at https://intuitive-robots.github.io/mdt_policy/.","url_abs":"https://arxiv.org/abs/2407.05996v1","url_pdf":"https://arxiv.org/pdf/2407.05996v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"multimodal-diffusion-transformer-learning","repo_url":"https://github.com/intuitive-robots/mdt_policy","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.05996","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.05996"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/intuitive-robots/mdt_policy","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran":14,"unverified":2},"by_repo_kind":{"official":{"samples":17,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"43f2167b019a5fb7","entry":"append_zero","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/models/edm_diffusion/gc_sampling.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/models/edm_diffusion/gc_sampling.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43f2167b019a5fb7"}},{"code_sha256_prefix":"907517ed3b614cc8","entry":"check_condition","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/evaluation/multistep_sequences.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/evaluation/multistep_sequences.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"907517ed3b614cc8"}},{"code_sha256_prefix":"924cae6f210421b2","entry":"count_success","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/evaluation/mdt_evaluate.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/evaluation/mdt_evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"924cae6f210421b2"}},{"code_sha256_prefix":"84f14e237dee0362","entry":"get_checkpoint_i_from_dir","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/evaluation/utils.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84f14e237dee0362"}},{"code_sha256_prefix":"e2bd842daf595f33","entry":"get_log_dir","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/evaluation/mdt_evaluate.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/evaluation/mdt_evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2bd842daf595f33"}},{"code_sha256_prefix":"205e97641ece09f8","entry":"get_sigmas_exponential","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/models/edm_diffusion/gc_sampling.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/models/edm_diffusion/gc_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"205e97641ece09f8"}},{"code_sha256_prefix":"a3ae2e8a897b4d99","entry":"get_sigmas_karras","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/models/edm_diffusion/gc_sampling.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/models/edm_diffusion/gc_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3ae2e8a897b4d99"}},{"code_sha256_prefix":"bc8ceb8ab2fa1d19","entry":"get_video_tag","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/evaluation/mdt_evaluate.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/evaluation/mdt_evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc8ceb8ab2fa1d19"}},{"code_sha256_prefix":"6cd960334bdf5b85","entry":"is_multi_gpu_training","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/training.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/training.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6cd960334bdf5b85"}},{"code_sha256_prefix":"30fcb27ec8c091af","entry":"load_class","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/evaluation/utils.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30fcb27ec8c091af"}},{"code_sha256_prefix":"c0de2c3f2828b5a3","entry":"load_npz","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/datasets/disk_dataset.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/datasets/disk_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0de2c3f2828b5a3"}},{"code_sha256_prefix":"3ffdb6fe27bf366d","entry":"load_pkl","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/datasets/disk_dataset.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/datasets/disk_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ffdb6fe27bf366d"}},{"code_sha256_prefix":"08c8831b3905ef52","entry":"setup_logger","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/training.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/training.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08c8831b3905ef52"}},{"code_sha256_prefix":"916d58d0838c488b","entry":"update_state","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/evaluation/multistep_sequences.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/evaluation/multistep_sequences.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"916d58d0838c488b"}},{"code_sha256_prefix":"5879a63c0a8f823e","entry":"valid_task","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/evaluation/multistep_sequences.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/evaluation/multistep_sequences.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5879a63c0a8f823e"}},{"code_sha256_prefix":"3e9e61cbcea37bd0","entry":"load_evaluation_checkpoint","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/evaluation/utils.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e9e61cbcea37bd0"}},{"code_sha256_prefix":"8ab9acda1d0ae3ee","entry":"setup_callbacks","repo":"intuitive-robots/mdt_policy","repo_kind":"official","path":"mdt/training.py","file_url":"https://github.com/intuitive-robots/mdt_policy/blob/HEAD/mdt/training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ab9acda1d0ae3ee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}