{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dreamtalk-when-expressive-talking-head","title":"DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Models","arxiv_id":"2312.09767","date":"2023-12-15","proceeding":null,"authors":["Yifeng Ma","Shiwei Zhang","Jiayu Wang","Xiang Wang","Yingya Zhang","Zhidong Deng"],"abstract":"Emotional talking head generation has attracted growing attention. Previous methods, which are mainly GAN-based, still struggle to consistently produce satisfactory results across diverse emotions and cannot conveniently specify personalized emotions. In this work, we leverage powerful diffusion models to address the issue and propose DreamTalk, a framework that employs meticulous design to unlock the potential of diffusion models in generating emotional talking heads. Specifically, DreamTalk consists of three crucial components: a denoising network, a style-aware lip expert, and a style predictor. The diffusion-based denoising network can consistently synthesize high-quality audio-driven face motions across diverse emotions. To enhance lip-motion accuracy and emotional fullness, we introduce a style-aware lip expert that can guide lip-sync while preserving emotion intensity. To more conveniently specify personalized emotions, a diffusion-based style predictor is utilized to predict the personalized emotion directly from the audio, eliminating the need for extra emotion reference. By this means, DreamTalk can consistently generate vivid talking faces across diverse emotions and conveniently specify personalized emotions. Extensive experiments validate DreamTalk's effectiveness and superiority. The code is available at https://github.com/ali-vilab/dreamtalk.","url_abs":"https://arxiv.org/abs/2312.09767v3","url_pdf":"https://arxiv.org/pdf/2312.09767v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dreamtalk-when-expressive-talking-head","repo_url":"https://github.com/ali-vilab/dreamtalk","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"talking-head-generation","task_name":"Talking Head Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.09767","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.09767"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ali-vilab/dreamtalk","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"393ccc99a3b65cf6","entry":"face3d_norm_to_raw","repo":"ali-vilab/dreamtalk","repo_kind":"official","path":"core/networks/diffusion_net.py","file_url":"https://github.com/ali-vilab/dreamtalk/blob/HEAD/core/networks/diffusion_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"393ccc99a3b65cf6"}},{"code_sha256_prefix":"7e7a2a7876ca493c","entry":"face3d_raw_to_norm","repo":"ali-vilab/dreamtalk","repo_kind":"official","path":"core/networks/diffusion_net.py","file_url":"https://github.com/ali-vilab/dreamtalk/blob/HEAD/core/networks/diffusion_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e7a2a7876ca493c"}},{"code_sha256_prefix":"58ed5133da1179a6","entry":"get_face3d_clip","repo":"ali-vilab/dreamtalk","repo_kind":"official","path":"core/utils.py","file_url":"https://github.com/ali-vilab/dreamtalk/blob/HEAD/core/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58ed5133da1179a6"}},{"code_sha256_prefix":"ff61b3c1dcddc54d","entry":"get_style_video_lists","repo":"ali-vilab/dreamtalk","repo_kind":"official","path":"core/utils.py","file_url":"https://github.com/ali-vilab/dreamtalk/blob/HEAD/core/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff61b3c1dcddc54d"}},{"code_sha256_prefix":"eff3585e52fa9c67","entry":"get_video_style","repo":"ali-vilab/dreamtalk","repo_kind":"official","path":"core/utils.py","file_url":"https://github.com/ali-vilab/dreamtalk/blob/HEAD/core/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eff3585e52fa9c67"}},{"code_sha256_prefix":"e8af53cde1de7240","entry":"mish","repo":"ali-vilab/dreamtalk","repo_kind":"official","path":"core/networks/mish.py","file_url":"https://github.com/ali-vilab/dreamtalk/blob/HEAD/core/networks/mish.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8af53cde1de7240"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}