{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tm2t-stochastic-and-tokenized-modeling-for","title":"TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts","arxiv_id":"2207.01696","date":"2022-07-04","proceeding":null,"authors":["Chuan Guo","Xinxin Zuo","Sen Wang","Li Cheng"],"abstract":"Inspired by the strong ties between vision and language, the two intimate human sensing and communication modalities, our paper aims to explore the generation of 3D human full-body motions from texts, as well as its reciprocal task, shorthanded for text2motion and motion2text, respectively. To tackle the existing challenges, especially to enable the generation of multiple distinct motions from the same text, and to avoid the undesirable production of trivial motionless pose sequences, we propose the use of motion token, a discrete and compact motion representation. This provides one level playing ground when considering both motions and text signals, as the motion and text tokens, respectively. Moreover, our motion2text module is integrated into the inverse alignment process of our text2motion training pipeline, where a significant deviation of synthesized text from the input text would be penalized by a large training loss; empirically this is shown to effectively improve performance. Finally, the mappings in-between the two modalities of motions and texts are facilitated by adapting the neural model for machine translation (NMT) to our context. This autoregressive modeling of the distribution over discrete motion tokens further enables non-deterministic production of pose sequences, of variable lengths, from an input text. Our approach is flexible, could be used for both text2motion and motion2text tasks. Empirical evaluations on two benchmark datasets demonstrate the superior performance of our approach on both tasks over a variety of state-of-the-art methods. Project page: https://ericguo5513.github.io/TM2T/","url_abs":"https://arxiv.org/abs/2207.01696v2","url_pdf":"https://arxiv.org/pdf/2207.01696v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tm2t-stochastic-and-tokenized-modeling-for","repo_url":"https://github.com/EricGuo5513/TM2T","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"motion-captioning","task_name":"Motion Captioning"},{"task_slug":"motion-synthesis","task_name":"Motion Synthesis"},{"task_slug":"nmt","task_name":"NMT"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/motion-captioning-on-humanml3d","task":"Motion Captioning","dataset":"HumanML3D","model":"TM2T","rank_in_archive_order":3,"of":4,"metrics":{"BERTScore":"37.8","BLEU-4":"22.3"},"uses_additional_data":false},{"leaderboard":"/sota/motion-captioning-on-kit-motion-language","task":"Motion Captioning","dataset":"KIT Motion-Language","model":"TM2T","rank_in_archive_order":3,"of":3,"metrics":{"BERTScore":"23.0","BLEU-4":"18.4"},"uses_additional_data":false},{"leaderboard":"/sota/motion-synthesis-on-humanml3d","task":"Motion Synthesis","dataset":"HumanML3D","model":"TM2T","rank_in_archive_order":34,"of":37,"metrics":{"Diversity":"8.589","FID":"1.501","Multimodality":"2.424","R Precision Top3":"0.729"},"uses_additional_data":false},{"leaderboard":"/sota/motion-synthesis-on-humanml3d","task":"Motion Synthesis","dataset":"HumanML3D","model":"Text2Gesture","rank_in_archive_order":35,"of":37,"metrics":{"Diversity":"6.409","FID":"5.012","R Precision Top3":"0.345"},"uses_additional_data":false},{"leaderboard":"/sota/motion-synthesis-on-humanml3d","task":"Motion Synthesis","dataset":"HumanML3D","model":"Language2Pose","rank_in_archive_order":37,"of":37,"metrics":{"Diversity":"7.676","FID":"11.02","R Precision Top3":"0.486"},"uses_additional_data":false},{"leaderboard":"/sota/motion-synthesis-on-kit-motion-language","task":"Motion Synthesis","dataset":"KIT Motion-Language","model":"TM2T","rank_in_archive_order":28,"of":31,"metrics":{"Diversity":"9.473","FID":"3.599","Multimodality":"3.292","R Precision Top3":"0.587"},"uses_additional_data":false},{"leaderboard":"/sota/motion-synthesis-on-kit-motion-language","task":"Motion Synthesis","dataset":"KIT Motion-Language","model":"Language2Pose","rank_in_archive_order":30,"of":31,"metrics":{"Diversity":"9.073","FID":"6.545","R Precision Top3":"0.483"},"uses_additional_data":false},{"leaderboard":"/sota/motion-synthesis-on-kit-motion-language","task":"Motion Synthesis","dataset":"KIT Motion-Language","model":"Text2Gesture","rank_in_archive_order":31,"of":31,"metrics":{"Diversity":"9.334","FID":"12.12","R Precision Top3":"0.338"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2207.01696","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.01696"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EricGuo5513/TM2T","reach":null}],"summary":{"ran":4,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"303be4aaa51b4ff2","entry":"AttLayer","repo":"EricGuo5513/TM2T","repo_kind":"official","path":"networks/modules.py","file_url":"https://github.com/EricGuo5513/TM2T/blob/HEAD/networks/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"303be4aaa51b4ff2"}},{"code_sha256_prefix":"bc63067f9f9cb05f","entry":"MotionEarlyAttDecoder","repo":"EricGuo5513/TM2T","repo_kind":"official","path":"networks/modules.py","file_url":"https://github.com/EricGuo5513/TM2T/blob/HEAD/networks/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc63067f9f9cb05f"}},{"code_sha256_prefix":"a7469c6b1b58f36a","entry":"MotionLateAttDecoder","repo":"EricGuo5513/TM2T","repo_kind":"official","path":"networks/modules.py","file_url":"https://github.com/EricGuo5513/TM2T/blob/HEAD/networks/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7469c6b1b58f36a"}},{"code_sha256_prefix":"b1963408ae788eed","entry":"TextEncoderBiGRU","repo":"EricGuo5513/TM2T","repo_kind":"official","path":"networks/modules.py","file_url":"https://github.com/EricGuo5513/TM2T/blob/HEAD/networks/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1963408ae788eed"}},{"code_sha256_prefix":"0f0021cdea13e4da","entry":"top_k_logits","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"0f0021cdea13e4da"}},{"code_sha256_prefix":"f7bb2a229205b3ff","entry":"Seq2SeqText2MotModel","repo":"EricGuo5513/TM2T","repo_kind":"official","path":"networks/modules.py","file_url":"https://github.com/EricGuo5513/TM2T/blob/HEAD/networks/modules.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7bb2a229205b3ff"}},{"code_sha256_prefix":"6bf2e0fd4ffbf8ff","entry":"init_weight","repo":"EricGuo5513/TM2T","repo_kind":"official","path":"networks/modules.py","file_url":"https://github.com/EricGuo5513/TM2T/blob/HEAD/networks/modules.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bf2e0fd4ffbf8ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}