{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-motion-refinement-for-unsupervised","title":"Learning Motion Refinement for Unsupervised Face Animation","arxiv_id":null,"date":"2023-09-21","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"Unsupervised face animation aims to generate a human face video based on the\nappearance of a source image, mimicking the motion from a driving video. Existing\nmethods typically adopted a prior-based motion model (e.g., the local affine motion\nmodel or the local thin-plate-spline motion model). While it is able to capture\nthe coarse facial motion, artifacts can often be observed around the tiny motion\nin local areas (e.g., lips and eyes), due to the limited ability of these methods\nto model the finer facial motions. In this work, we design a new unsupervised\nface animation approach to learn simultaneously the coarse and finer motions. In\nparticular, while exploiting the local affine motion model to learn the global coarse\nfacial motion, we design a novel motion refinement module to compensate for\nthe local affine motion model for modeling finer face motions in local areas. The\nmotion refinement is learned from the dense correlation between the source and\ndriving images. Specifically, we first construct a structure correlation volume based\non the keypoint features of the source and driving images. Then, we train a model\nto generate the tiny facial motions iteratively from low to high resolution. The\nlearned motion refinements are combined with the coarse motion to generate the\nnew image. Extensive experiments on widely used benchmarks demonstrate that\nour method achieves the best results among state-of-the-art baselines.","url_abs":"https://openreview.net/forum?id=m9uHv1Pxq7","url_pdf":"https://openreview.net/pdf?id=m9uHv1Pxq7","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-motion-refinement-for-unsupervised","repo_url":"https://github.com/jialetao/mrfa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}