{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/livelyspeaker-towards-semantic-aware-co","title":"LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation","arxiv_id":"2309.09294","date":"2023-09-17","proceeding":"ICCV 2023 1","authors":["YiHao Zhi","Xiaodong Cun","Xuelin Chen","Xi Shen","Wen Guo","Shaoli Huang","Shenghua Gao"],"abstract":"Gestures are non-verbal but important behaviors accompanying people's speech. While previous methods are able to generate speech rhythm-synchronized gestures, the semantic context of the speech is generally lacking in the gesticulations. Although semantic gestures do not occur very regularly in human speech, they are indeed the key for the audience to understand the speech context in a more immersive environment. Hence, we introduce LivelySpeaker, a framework that realizes semantics-aware co-speech gesture generation and offers several control handles. In particular, our method decouples the task into two stages: script-based gesture generation and audio-guided rhythm refinement. Specifically, the script-based gesture generation leverages the pre-trained CLIP text embeddings as the guidance for generating gestures that are highly semantically aligned with the script. Then, we devise a simple but effective diffusion-based gesture generation backbone simply using pure MLPs, that is conditioned on only audio signals and learns to gesticulate with realistic motions. We utilize such powerful prior to rhyme the script-guided gestures with the audio signals, notably in a zero-shot setting. Our novel two-stage generation framework also enables several applications, such as changing the gesticulation style, editing the co-speech gestures via textual prompting, and controlling the semantic awareness and rhythm alignment with guided diffusion. Extensive experiments demonstrate the advantages of the proposed framework over competing methods. In addition, our core diffusion-based generative model also achieves state-of-the-art performance on two benchmarks. The code and model will be released to facilitate future research.","url_abs":"https://arxiv.org/abs/2309.09294v1","url_pdf":"https://arxiv.org/pdf/2309.09294v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"livelyspeaker-towards-semantic-aware-co","repo_url":"https://github.com/zyhbili/livelyspeaker","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"gesture-generation","task_name":"Gesture Generation"},{"task_slug":"rhythm","task_name":"Rhythm"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.09294","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.09294"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zyhbili/livelyspeaker","reach":null}],"summary":{"ran":7,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"ab10e3b84a1731cf","entry":"InputProcess","repo":"zyhbili/livelyspeaker","repo_kind":"official","path":"scripts_beat/model/RAG.py","file_url":"https://github.com/zyhbili/livelyspeaker/blob/HEAD/scripts_beat/model/RAG.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab10e3b84a1731cf"}},{"code_sha256_prefix":"94b3e0cc7ad2ed65","entry":"LN_spatial","repo":"zyhbili/livelyspeaker","repo_kind":"official","path":"scripts_beat/model/RAG.py","file_url":"https://github.com/zyhbili/livelyspeaker/blob/HEAD/scripts_beat/model/RAG.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94b3e0cc7ad2ed65"}},{"code_sha256_prefix":"135719f36af53f65","entry":"MLPblock","repo":"zyhbili/livelyspeaker","repo_kind":"official","path":"scripts_beat/model/RAG.py","file_url":"https://github.com/zyhbili/livelyspeaker/blob/HEAD/scripts_beat/model/RAG.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"135719f36af53f65"}},{"code_sha256_prefix":"5f6f099d18a813b0","entry":"OutputProcess","repo":"zyhbili/livelyspeaker","repo_kind":"official","path":"scripts_beat/model/RAG.py","file_url":"https://github.com/zyhbili/livelyspeaker/blob/HEAD/scripts_beat/model/RAG.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f6f099d18a813b0"}},{"code_sha256_prefix":"1e4be20e6926eb5e","entry":"PositionalEncoding","repo":"zyhbili/livelyspeaker","repo_kind":"official","path":"scripts_beat/model/RAG.py","file_url":"https://github.com/zyhbili/livelyspeaker/blob/HEAD/scripts_beat/model/RAG.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e4be20e6926eb5e"}},{"code_sha256_prefix":"99a7bc771324d5c6","entry":"TimestepEmbedder","repo":"zyhbili/livelyspeaker","repo_kind":"official","path":"scripts_beat/model/RAG.py","file_url":"https://github.com/zyhbili/livelyspeaker/blob/HEAD/scripts_beat/model/RAG.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99a7bc771324d5c6"}},{"code_sha256_prefix":"4ebadac6d7d14e1e","entry":"TransMLP","repo":"zyhbili/livelyspeaker","repo_kind":"official","path":"scripts_beat/model/RAG.py","file_url":"https://github.com/zyhbili/livelyspeaker/blob/HEAD/scripts_beat/model/RAG.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ebadac6d7d14e1e"}},{"code_sha256_prefix":"933355f8e7b3b4b1","entry":"build_mlps","repo":"zyhbili/livelyspeaker","repo_kind":"official","path":"scripts_beat/model/RAG.py","file_url":"https://github.com/zyhbili/livelyspeaker/blob/HEAD/scripts_beat/model/RAG.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"933355f8e7b3b4b1"}},{"code_sha256_prefix":"02f5cb772a1e8068","entry":"RAG","repo":"zyhbili/livelyspeaker","repo_kind":"official","path":"scripts_beat/model/RAG.py","file_url":"https://github.com/zyhbili/livelyspeaker/blob/HEAD/scripts_beat/model/RAG.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"02f5cb772a1e8068"}},{"code_sha256_prefix":"924d11fd0a7f4a5b","entry":"WavEncoder","repo":"zyhbili/livelyspeaker","repo_kind":"official","path":"scripts_beat/model/RAG.py","file_url":"https://github.com/zyhbili/livelyspeaker/blob/HEAD/scripts_beat/model/RAG.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"924d11fd0a7f4a5b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}