{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/m-3-gpt-an-advanced-multimodal-multitask","title":"M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation","arxiv_id":"2405.16273","date":"2024-05-25","proceeding":null,"authors":["Mingshuang Luo","Ruibing Hou","Zhuo Li","Hong Chang","Zimo Liu","YaoWei Wang","Shiguang Shan"],"abstract":"This paper presents M$^3$GPT, an advanced $\\textbf{M}$ultimodal, $\\textbf{M}$ultitask framework for $\\textbf{M}$otion comprehension and generation. M$^3$GPT operates on three fundamental principles. The first focuses on creating a unified representation space for various motion-relevant modalities. We employ discrete vector quantization for multimodal conditional signals, such as text, music and motion/dance, enabling seamless integration into a large language model (LLM) with a single vocabulary. The second involves modeling motion generation directly in the raw motion space. This strategy circumvents the information loss associated with a discrete tokenizer, resulting in more detailed and comprehensive motion generation. Third, M$^3$GPT learns to model the connections and synergies among various motion-relevant tasks. Text, the most familiar and well-understood modality for LLMs, is utilized as a bridge to establish connections between different motion tasks, facilitating mutual reinforcement. To our knowledge, M$^3$GPT is the first model capable of comprehending and generating motions based on multiple signals. Extensive experiments highlight M$^3$GPT's superior performance across various motion-relevant tasks and its powerful zero-shot generalization capabilities for extremely challenging tasks. Project page: \\url{https://github.com/luomingshuang/M3GPT}.","url_abs":"https://arxiv.org/abs/2405.16273v5","url_pdf":"https://arxiv.org/pdf/2405.16273v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"m-3-gpt-an-advanced-multimodal-multitask","repo_url":"https://github.com/luomingshuang/m3gpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"motion-generation","task_name":"Motion Generation"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.16273","atlas_url":"https://app.syntology.ai/?focus=2405.16273","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.16273"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/luomingshuang/M3GPT","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luomingshuang/m3gpt","reach":{"status":"ok"}}],"summary":{"ran":15,"ran_draft_wrong":2,"unverified":8},"by_repo_kind":{"official":{"samples":25,"ran":17,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":25,"samples":[{"code_sha256_prefix":"af4cc52db9e5aa3b","entry":"Conv2d","repo":"luomingshuang/m3gpt","repo_kind":"official","path":"m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","file_url":"https://github.com/luomingshuang/m3gpt/blob/HEAD/m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af4cc52db9e5aa3b"}},{"code_sha256_prefix":"393e561d1764ea4e","entry":"CrossAttentionLayer","repo":"luomingshuang/m3gpt","repo_kind":"official","path":"m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","file_url":"https://github.com/luomingshuang/m3gpt/blob/HEAD/m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"393e561d1764ea4e"}},{"code_sha256_prefix":"dddec67cab845dc9","entry":"FFNLayer","repo":"luomingshuang/m3gpt","repo_kind":"official","path":"m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","file_url":"https://github.com/luomingshuang/m3gpt/blob/HEAD/m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dddec67cab845dc9"}},{"code_sha256_prefix":"cb65f982177791b7","entry":"PositionEmbeddingSine","repo":"luomingshuang/m3gpt","repo_kind":"official","path":"m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","file_url":"https://github.com/luomingshuang/m3gpt/blob/HEAD/m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb65f982177791b7"}},{"code_sha256_prefix":"cdc9f7050bfb9dbf","entry":"SelfAttentionLayer","repo":"luomingshuang/m3gpt","repo_kind":"official","path":"m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","file_url":"https://github.com/luomingshuang/m3gpt/blob/HEAD/m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cdc9f7050bfb9dbf"}},{"code_sha256_prefix":"78d1f9f766be74b7","entry":"accuracy","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/utils.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"78d1f9f766be74b7"}},{"code_sha256_prefix":"8a63c5da44344ba6","entry":"accuracy_multi","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/utils.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a63c5da44344ba6"}},{"code_sha256_prefix":"4570d6d70bccdfb6","entry":"flat","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/config.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4570d6d70bccdfb6"}},{"code_sha256_prefix":"bd1cba25bacbf54d","entry":"get_logger","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/utils.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd1cba25bacbf54d"}},{"code_sha256_prefix":"eb8a2600afc1de95","entry":"init","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/fp16/amp.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/fp16/amp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb8a2600afc1de95"}},{"code_sha256_prefix":"4418374c5f2c445c","entry":"mask2pos","repo":"luomingshuang/m3gpt","repo_kind":"official","path":"m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","file_url":"https://github.com/luomingshuang/m3gpt/blob/HEAD/m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4418374c5f2c445c"}},{"code_sha256_prefix":"70165e0988ce9571","entry":"param_group_fc","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/make_param_group.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/make_param_group.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70165e0988ce9571"}},{"code_sha256_prefix":"e4a7663d686ce91a","entry":"param_group_multitask","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/make_param_group.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/make_param_group.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4a7663d686ce91a"}},{"code_sha256_prefix":"0540cf098dbc3d42","entry":"param_group_no_wd","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/make_param_group.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/make_param_group.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0540cf098dbc3d42"}},{"code_sha256_prefix":"5d5d46e55c97947a","entry":"pos2posemb2d","repo":"luomingshuang/m3gpt","repo_kind":"official","path":"m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","file_url":"https://github.com/luomingshuang/m3gpt/blob/HEAD/m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d5d46e55c97947a"}},{"code_sha256_prefix":"7d6e263caea8e963","entry":"read_json","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/utils.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d6e263caea8e963"}},{"code_sha256_prefix":"ff1253be17efe1ae","entry":"retry_if_cuda_oom","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/memory.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/memory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff1253be17efe1ae"}},{"code_sha256_prefix":"e2a72e23375812ed","entry":"TransformerDecoder","repo":"luomingshuang/m3gpt","repo_kind":"official","path":"m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","file_url":"https://github.com/luomingshuang/m3gpt/blob/HEAD/m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2a72e23375812ed"}},{"code_sha256_prefix":"fe1c1ba4ad057e5f","entry":"c2_xavier_fill","repo":"luomingshuang/m3gpt","repo_kind":"official","path":"m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","file_url":"https://github.com/luomingshuang/m3gpt/blob/HEAD/m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe1c1ba4ad057e5f"}},{"code_sha256_prefix":"c7d1fdb02e6cad00","entry":"dist_init","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/distributed_utils.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/distributed_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7d1fdb02e6cad00"}},{"code_sha256_prefix":"f27c7ab614e20039","entry":"float_function","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/fp16/amp.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/fp16/amp.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f27c7ab614e20039"}},{"code_sha256_prefix":"09d724b291a01fbe","entry":"get_2d_sincos_pos_embed","repo":"luomingshuang/m3gpt","repo_kind":"official","path":"m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","file_url":"https://github.com/luomingshuang/m3gpt/blob/HEAD/m3gpt/core/models/decoders/network/transformer_decoder/transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09d724b291a01fbe"}},{"code_sha256_prefix":"6b4eee4b4bb1f238","entry":"half_function","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/fp16/amp.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/fp16/amp.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b4eee4b4bb1f238"}},{"code_sha256_prefix":"b1471eccec180d87","entry":"simple_group_split","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/distributed_utils.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/distributed_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1471eccec180d87"}},{"code_sha256_prefix":"4e36f197c125d4d3","entry":"specific_group_split","repo":"luomingshuang/M3GPT","repo_kind":"official","path":"m3gpt/core/distributed_utils.py","file_url":"https://github.com/luomingshuang/M3GPT/blob/HEAD/m3gpt/core/distributed_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e36f197c125d4d3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}