{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/qpgesture-quantization-based-and-phase-guided-1","title":"QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation","arxiv_id":"2305.11094","date":"2023-05-18","proceeding":"CVPR 2023 1","authors":["Sicheng Yang","Zhiyong Wu","Minglei Li","Zhensong Zhang","Lei Hao","Weihong Bao","Haolin Zhuang"],"abstract":"Speech-driven gesture generation is highly challenging due to the random jitters of human motion. In addition, there is an inherent asynchronous relationship between human speech and gestures. To tackle these challenges, we introduce a novel quantization-based and phase-guided motion-matching framework. Specifically, we first present a gesture VQ-VAE module to learn a codebook to summarize meaningful gesture units. With each code representing a unique gesture, random jittering problems are alleviated effectively. We then use Levenshtein distance to align diverse gestures with different speech. Levenshtein distance based on audio quantization as a similarity metric of corresponding speech of gestures helps match more appropriate gestures with speech, and solves the alignment problem of speech and gestures well. Moreover, we introduce phase to guide the optimal gesture matching based on the semantics of context or rhythm of audio. Phase guides when text-based or speech-based gestures should be performed to make the generated gestures more natural. Extensive experiments show that our method outperforms recent approaches on speech-driven gesture generation. Our code, database, pre-trained models, and demos are available at https://github.com/YoungSeng/QPGesture.","url_abs":"https://arxiv.org/abs/2305.11094v1","url_pdf":"https://arxiv.org/pdf/2305.11094v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"qpgesture-quantization-based-and-phase-guided-1","repo_url":"https://github.com/youngseng/qpgesture","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"gesture-generation","task_name":"Gesture Generation"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"rhythm","task_name":"Rhythm"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"vq-vae","method_name":"VQ-VAE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.11094","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.11094"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/YoungSeng/QPGesture","reach":null}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"a9febf7c3f3916cd","entry":"Encoder","repo":"YoungSeng/QPGesture","repo_kind":"official","path":"codebook/models/simpleVqvae.py","file_url":"https://github.com/YoungSeng/QPGesture/blob/HEAD/codebook/models/simpleVqvae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9febf7c3f3916cd"}},{"code_sha256_prefix":"a5828a3b879de771","entry":"LinearNorm","repo":"YoungSeng/QPGesture","repo_kind":"official","path":"codebook/models/simpleVqvae.py","file_url":"https://github.com/YoungSeng/QPGesture/blob/HEAD/codebook/models/simpleVqvae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5828a3b879de771"}},{"code_sha256_prefix":"8126d0000831ce5c","entry":"VQEmbeddingEMA","repo":"YoungSeng/QPGesture","repo_kind":"official","path":"codebook/models/simpleVqvae.py","file_url":"https://github.com/YoungSeng/QPGesture/blob/HEAD/codebook/models/simpleVqvae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"none","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8126d0000831ce5c"}},{"code_sha256_prefix":"aaa95e7d61da5663","entry":"Decoder","repo":"YoungSeng/QPGesture","repo_kind":"official","path":"codebook/models/simpleVqvae.py","file_url":"https://github.com/YoungSeng/QPGesture/blob/HEAD/codebook/models/simpleVqvae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aaa95e7d61da5663"}},{"code_sha256_prefix":"8176f9f8d472ffb7","entry":"simpleVQVAE","repo":"YoungSeng/QPGesture","repo_kind":"official","path":"codebook/models/simpleVqvae.py","file_url":"https://github.com/YoungSeng/QPGesture/blob/HEAD/codebook/models/simpleVqvae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8176f9f8d472ffb7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}