{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/motionllm-multimodal-motion-language-learning","title":"Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs","arxiv_id":"2405.17013","date":"2024-05-27","proceeding":null,"authors":["Qi Wu","Yubo Zhao","Yifan Wang","Xinhang Liu","Yu-Wing Tai","Chi-Keung Tang"],"abstract":"While previous approaches to 3D human motion generation have achieved notable success, they often rely on extensive training and are limited to specific tasks. To address these challenges, we introduce Motion-Agent, an efficient conversational framework designed for general human motion generation, editing, and understanding. Motion-Agent employs an open-source pre-trained language model to develop a generative agent, MotionLLM, that bridges the gap between motion and text. This is accomplished by encoding and quantizing motions into discrete tokens that align with the language model's vocabulary. With only 1--3\\% of the model's parameters fine-tuned using adapters, MotionLLM delivers performance on par with diffusion models and other transformer-based methods trained from scratch. By integrating MotionLLM with GPT-4 without additional training, Motion-Agent is able to generate highly complex motion sequences through multi-turn conversations, a capability that previous models have struggled to achieve. Motion-Agent supports a wide range of motion-language tasks, offering versatile capabilities for generating and customizing human motion through interactive conversational exchanges. Project page: https://knoxzhao.github.io/Motion-Agent","url_abs":"https://arxiv.org/abs/2405.17013v3","url_pdf":"https://arxiv.org/pdf/2405.17013v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"motionllm-multimodal-motion-language-learning","repo_url":"https://github.com/szqwu/Motion-Agent","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"motion-captioning","task_name":"Motion Captioning"},{"task_slug":"motion-generation","task_name":"Motion Generation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.17013","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.17013"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/szqwu/Motion-Agent","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d09da8be655b6b18","entry":"build_one_instance_m2t","repo":"szqwu/Motion-Agent","repo_kind":"official","path":"models/training_utils.py","file_url":"https://github.com/szqwu/Motion-Agent/blob/HEAD/models/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d09da8be655b6b18"}},{"code_sha256_prefix":"d45be01eab6f74eb","entry":"build_one_instance_t2m","repo":"szqwu/Motion-Agent","repo_kind":"official","path":"models/training_utils.py","file_url":"https://github.com/szqwu/Motion-Agent/blob/HEAD/models/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d45be01eab6f74eb"}},{"code_sha256_prefix":"39d21578902eb9d4","entry":"process_batch","repo":"szqwu/Motion-Agent","repo_kind":"official","path":"models/training_utils.py","file_url":"https://github.com/szqwu/Motion-Agent/blob/HEAD/models/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39d21578902eb9d4"}},{"code_sha256_prefix":"9593fec6c1bf25e7","entry":"qinv","repo":"szqwu/Motion-Agent","repo_kind":"official","path":"utils/motion_utils.py","file_url":"https://github.com/szqwu/Motion-Agent/blob/HEAD/utils/motion_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9593fec6c1bf25e7"}},{"code_sha256_prefix":"197ed23772e8e0c6","entry":"qrot","repo":"szqwu/Motion-Agent","repo_kind":"official","path":"utils/motion_utils.py","file_url":"https://github.com/szqwu/Motion-Agent/blob/HEAD/utils/motion_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"197ed23772e8e0c6"}},{"code_sha256_prefix":"5ad104298e7e0eff","entry":"recover_root_rot_pos","repo":"szqwu/Motion-Agent","repo_kind":"official","path":"utils/motion_utils.py","file_url":"https://github.com/szqwu/Motion-Agent/blob/HEAD/utils/motion_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ad104298e7e0eff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}