{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/move-as-you-say-interact-as-you-can-language","title":"Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance","arxiv_id":"2403.18036","date":"2024-03-26","proceeding":null,"authors":["Zan Wang","Yixin Chen","Baoxiong Jia","Puhao Li","Jinlu Zhang","Jingze Zhang","Tengyu Liu","Yixin Zhu","Wei Liang","Siyuan Huang"],"abstract":"Despite significant advancements in text-to-motion synthesis, generating language-guided human motion within 3D environments poses substantial challenges. These challenges stem primarily from (i) the absence of powerful generative models capable of jointly modeling natural language, 3D scenes, and human motion, and (ii) the generative models' intensive data requirements contrasted with the scarcity of comprehensive, high-quality, language-scene-motion datasets. To tackle these issues, we introduce a novel two-stage framework that employs scene affordance as an intermediate representation, effectively linking 3D scene grounding and conditional motion generation. Our framework comprises an Affordance Diffusion Model (ADM) for predicting explicit affordance map and an Affordance-to-Motion Diffusion Model (AMDM) for generating plausible human motions. By leveraging scene affordance maps, our method overcomes the difficulty in generating human motion under multimodal condition signals, especially when training with limited data lacking extensive language-scene-motion pairs. Our extensive experiments demonstrate that our approach consistently outperforms all baselines on established benchmarks, including HumanML3D and HUMANISE. Additionally, we validate our model's exceptional generalization capabilities on a specially curated evaluation set featuring previously unseen descriptions and scenes.","url_abs":"https://arxiv.org/abs/2403.18036v1","url_pdf":"https://arxiv.org/pdf/2403.18036v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"move-as-you-say-interact-as-you-can-language","repo_url":"https://github.com/afford-motion/afford-motion","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"motion-generation","task_name":"Motion Generation"},{"task_slug":"motion-synthesis","task_name":"Motion Synthesis"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.18036","atlas_url":"https://app.syntology.ai/?focus=2403.18036","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.18036"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/afford-motion/afford-motion","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/GuyTevet/motion-diffusion-model","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":5,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cfd76fd0d89574a4","entry":"approx_standard_normal_cdf","repo":"afford-motion/afford-motion","repo_kind":"official","path":"diffusion/losses.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/diffusion/losses.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfd76fd0d89574a4"}},{"code_sha256_prefix":"51f957c05d5376a6","entry":"collate_fn_general","repo":"afford-motion/afford-motion","repo_kind":"official","path":"datasets/misc.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/datasets/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51f957c05d5376a6"}},{"code_sha256_prefix":"cd33283d615fb3d7","entry":"discretized_gaussian_log_likelihood","repo":"afford-motion/afford-motion","repo_kind":"official","path":"diffusion/losses.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/diffusion/losses.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd33283d615fb3d7"}},{"code_sha256_prefix":"f804ab22c5a75066","entry":"get_positional_encoding","repo":"afford-motion/afford-motion","repo_kind":"official","path":"models/modules.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/models/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f804ab22c5a75066"}},{"code_sha256_prefix":"cf2798b666b231ca","entry":"normal_kl","repo":"afford-motion/afford-motion","repo_kind":"official","path":"diffusion/losses.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/diffusion/losses.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf2798b666b231ca"}},{"code_sha256_prefix":"9593fec6c1bf25e7","entry":"qinv","repo":"afford-motion/afford-motion","repo_kind":"official","path":"visualize_h3d.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/visualize_h3d.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9593fec6c1bf25e7"}},{"code_sha256_prefix":"197ed23772e8e0c6","entry":"qrot","repo":"afford-motion/afford-motion","repo_kind":"official","path":"visualize_h3d.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/visualize_h3d.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"197ed23772e8e0c6"}},{"code_sha256_prefix":"b71d6fae8fc956e6","entry":"recover_root_rot_pos","repo":"afford-motion/afford-motion","repo_kind":"official","path":"visualize_h3d.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/visualize_h3d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b71d6fae8fc956e6"}},{"code_sha256_prefix":"444a2a4cb3a6a1e5","entry":"collate_fn_squeeze_pcd_batch","repo":"afford-motion/afford-motion","repo_kind":"official","path":"datasets/misc.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/datasets/misc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"444a2a4cb3a6a1e5"}},{"code_sha256_prefix":"b6fdefe1544cf00e","entry":"create_gaussian_diffusion","repo":"afford-motion/afford-motion","repo_kind":"official","path":"models/base.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/models/base.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6fdefe1544cf00e"}},{"code_sha256_prefix":"67c06955f529f539","entry":"encode_text_bert","repo":"afford-motion/afford-motion","repo_kind":"official","path":"models/functions.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/models/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67c06955f529f539"}},{"code_sha256_prefix":"fd5c66a805ae04f0","entry":"load_and_freeze_bert_model","repo":"afford-motion/afford-motion","repo_kind":"official","path":"models/functions.py","file_url":"https://github.com/afford-motion/afford-motion/blob/HEAD/models/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd5c66a805ae04f0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}