{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/co-speech-gesture-video-generation-via-motion","title":"Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model","arxiv_id":"2404.01862","date":"2024-04-02","proceeding":"CVPR 2024 1","authors":["Xu He","Qiaochu Huang","Zhensong Zhang","Zhiwei Lin","Zhiyong Wu","Sicheng Yang","Minglei Li","Zhiyi Chen","Songcen Xu","Xiaofei Wu"],"abstract":"Co-speech gestures, if presented in the lively form of videos, can achieve superior visual effects in human-machine interaction. While previous works mostly generate structural human skeletons, resulting in the omission of appearance information, we focus on the direct generation of audio-driven co-speech gesture videos in this work. There are two main challenges: 1) A suitable motion feature is needed to describe complex human movements with crucial appearance information. 2) Gestures and speech exhibit inherent dependencies and should be temporally aligned even of arbitrary length. To solve these problems, we present a novel motion-decoupled framework to generate co-speech gesture videos. Specifically, we first introduce a well-designed nonlinear TPS transformation to obtain latent motion features preserving essential appearance information. Then a transformer-based diffusion model is proposed to learn the temporal correlation between gestures and speech, and performs generation in the latent motion space, followed by an optimal motion selection module to produce long-term coherent and consistent gesture videos. For better visual perception, we further design a refinement network focusing on missing details of certain areas. Extensive experimental results show that our proposed framework significantly outperforms existing approaches in both motion and video-related evaluations. Our code, demos, and more resources are available at https://github.com/thuhcsi/S2G-MDDiffusion.","url_abs":"https://arxiv.org/abs/2404.01862v1","url_pdf":"https://arxiv.org/pdf/2404.01862v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"co-speech-gesture-video-generation-via-motion","repo_url":"https://github.com/thuhcsi/s2g-mddiffusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.01862","atlas_url":"https://app.syntology.ai/?focus=2404.01862","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.01862"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/thuhcsi/S2G-MDDiffusion","reach":null}],"summary":{"ran_honours":2,"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"7f1040f5e3991d5e","entry":"identity","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"7f1040f5e3991d5e"}},{"code_sha256_prefix":"01c11cf700d39c20","entry":"EMA","repo":"thuhcsi/S2G-MDDiffusion","repo_kind":"official","path":"inference/model/diffusion.py","file_url":"https://github.com/thuhcsi/S2G-MDDiffusion/blob/HEAD/inference/model/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01c11cf700d39c20"}},{"code_sha256_prefix":"af34e5e7bd22eecd","entry":"GestureDiffusion","repo":"thuhcsi/S2G-MDDiffusion","repo_kind":"official","path":"inference/model/diffusion.py","file_url":"https://github.com/thuhcsi/S2G-MDDiffusion/blob/HEAD/inference/model/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af34e5e7bd22eecd"}},{"code_sha256_prefix":"c984209a2c66afbe","entry":"make_beta_schedule","repo":"thuhcsi/S2G-MDDiffusion","repo_kind":"official","path":"inference/model/diffusion.py","file_url":"https://github.com/thuhcsi/S2G-MDDiffusion/blob/HEAD/inference/model/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c984209a2c66afbe"}},{"code_sha256_prefix":"09c8479d9a5b3e06","entry":"extract","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"09c8479d9a5b3e06"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}