{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/humantomato-text-aligned-whole-body-motion","title":"HumanTOMATO: Text-aligned Whole-body Motion Generation","arxiv_id":"2310.12978","date":"2023-10-19","proceeding":null,"authors":["Shunlin Lu","Ling-Hao Chen","Ailing Zeng","Jing Lin","Ruimao Zhang","Lei Zhang","Heung-Yeung Shum"],"abstract":"This work targets a novel text-driven whole-body motion generation task, which takes a given textual description as input and aims at generating high-quality, diverse, and coherent facial expressions, hand gestures, and body motions simultaneously. Previous works on text-driven motion generation tasks mainly have two limitations: they ignore the key role of fine-grained hand and face controlling in vivid whole-body motion generation, and lack a good alignment between text and motion. To address such limitations, we propose a Text-aligned whOle-body Motion generATiOn framework, named HumanTOMATO, which is the first attempt to our knowledge towards applicable holistic motion generation in this research area. To tackle this challenging task, our solution includes two key designs: (1) a Holistic Hierarchical VQ-VAE (aka H$^2$VQ) and a Hierarchical-GPT for fine-grained body and hand motion reconstruction and generation with two structured codebooks; and (2) a pre-trained text-motion-alignment model to help generated motion align with the input textual description explicitly. Comprehensive experiments verify that our model has significant advantages in both the quality of generated motions and their alignment with text.","url_abs":"https://arxiv.org/abs/2310.12978v1","url_pdf":"https://arxiv.org/pdf/2310.12978v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"humantomato-text-aligned-whole-body-motion","repo_url":"https://github.com/IDEA-Research/HumanTOMATO","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"motion-generation","task_name":"Motion Generation"},{"task_slug":"motion-synthesis","task_name":"Motion Synthesis"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"vq-vae","method_name":"VQ-VAE"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/motion-synthesis-on-motion-x","task":"Motion Synthesis","dataset":"Motion-X","model":"HumanTOMATO","rank_in_archive_order":1,"of":4,"metrics":{"Diversity":"10.812","FID":"1.174","MModality":"1.732","TMR-Matching Score":"0.809","TMR-R-Precision Top3":"0.703"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.12978","atlas_url":"https://app.syntology.ai/?focus=2310.12978","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.12978"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/IDEA-Research/HumanTOMATO","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":12,"unverified":6},"by_repo_kind":{"official":{"samples":18,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"d1b34cb3b09bd220","entry":"findAllFile","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/dataset.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d1b34cb3b09bd220"}},{"code_sha256_prefix":"9013a28a43aa4fb6","entry":"get_model","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"OpenTMA/tma/models/get_model.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/OpenTMA/tma/models/get_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9013a28a43aa4fb6"}},{"code_sha256_prefix":"5acf2cc2e4bad9a2","entry":"get_module","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"OpenTMA/tma/models/get_model.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/OpenTMA/tma/models/get_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5acf2cc2e4bad9a2"}},{"code_sha256_prefix":"dafe32d9bbf4ede8","entry":"get_module_config","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"OpenTMA/tma/config.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/OpenTMA/tma/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dafe32d9bbf4ede8"}},{"code_sha256_prefix":"4607de5098f9dd03","entry":"get_obj_from_str","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"OpenTMA/tma/config.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/OpenTMA/tma/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4607de5098f9dd03"}},{"code_sha256_prefix":"1baed29ece06b185","entry":"instantiate_from_config","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"OpenTMA/tma/config.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/OpenTMA/tma/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1baed29ece06b185"}},{"code_sha256_prefix":"b5a772e4b068dd3a","entry":"mld_collate","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/dataset.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b5a772e4b068dd3a"}},{"code_sha256_prefix":"2c7add0e79b82aa9","entry":"neg_recall","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"OpenTMA/retrieval.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/OpenTMA/retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2c7add0e79b82aa9"}},{"code_sha256_prefix":"d9e44ea7fa11475a","entry":"process_smplx_322_data","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/smplx2joints.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/smplx2joints.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d9e44ea7fa11475a"}},{"code_sha256_prefix":"9593fec6c1bf25e7","entry":"qinv","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/common/quaternion.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/common/quaternion.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9593fec6c1bf25e7"}},{"code_sha256_prefix":"66024edafa8c53fb","entry":"qinv_np","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/common/quaternion.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/common/quaternion.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66024edafa8c53fb"}},{"code_sha256_prefix":"bac0877915201be1","entry":"qnormalize","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/common/quaternion.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/common/quaternion.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bac0877915201be1"}},{"code_sha256_prefix":"0b9f005ecb737755","entry":"collate_tensors","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/dataset.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b9f005ecb737755"}},{"code_sha256_prefix":"73db80d6e56c84b8","entry":"plot_3d_motion","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/plot_3d_global.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/plot_3d_global.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"73db80d6e56c84b8"}},{"code_sha256_prefix":"b657f9ab7a22fdfa","entry":"plot_3d_motion_smplh","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/plot_3d_global.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/plot_3d_global.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b657f9ab7a22fdfa"}},{"code_sha256_prefix":"484732998bd7f9fd","entry":"plot_3d_motion_smplh","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/plot_feature.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/plot_feature.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"484732998bd7f9fd"}},{"code_sha256_prefix":"fa243f2b8cc96a9f","entry":"recover_from_ric","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/plot_feature.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/plot_feature.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fa243f2b8cc96a9f"}},{"code_sha256_prefix":"abc992b18808701a","entry":"recover_root_rot_pos","repo":"IDEA-Research/HumanTOMATO","repo_kind":"official","path":"src/tomato_represenation/plot_feature.py","file_url":"https://github.com/IDEA-Research/HumanTOMATO/blob/HEAD/src/tomato_represenation/plot_feature.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"abc992b18808701a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}