{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-appearance-models-for-face-rendering","title":"Deep Appearance Models for Face Rendering","arxiv_id":"1808.00362","date":"2018-08-01","proceeding":null,"authors":["Stephen Lombardi","Jason Saragih","Tomas Simon","Yaser Sheikh"],"abstract":"We introduce a deep appearance model for rendering the human face. Inspired\nby Active Appearance Models, we develop a data-driven rendering pipeline that\nlearns a joint representation of facial geometry and appearance from a\nmultiview capture setup. Vertex positions and view-specific textures are\nmodeled using a deep variational autoencoder that captures complex nonlinear\neffects while producing a smooth and compact latent representation.\nView-specific texture enables the modeling of view-dependent effects such as\nspecularity. In addition, it can also correct for imperfect geometry stemming\nfrom biased or low resolution estimates. This is a significant departure from\nthe traditional graphics pipeline, which requires highly accurate geometry as\nwell as all elements of the shading model to achieve realism through\nphysically-inspired light transport. Acquiring such a high level of accuracy is\ndifficult in practice, especially for complex and intricate parts of the face,\nsuch as eyelashes and the oral cavity. These are handled naturally by our\napproach, which does not rely on precise estimates of geometry. Instead, the\nshading model accommodates deficiencies in geometry though the flexibility\nafforded by the neural network employed. At inference time, we condition the\ndecoding network on the viewpoint of the camera in order to generate the\nappropriate texture for rendering. The resulting system can be implemented\nsimply using existing rendering engines through dynamic textures with flat\nlighting. This representation, together with a novel unsupervised technique for\nmapping images to facial states, results in a system that is naturally suited\nto real-time interactive settings such as Virtual Reality (VR).","url_abs":"http://arxiv.org/abs/1808.00362v1","url_pdf":"http://arxiv.org/pdf/1808.00362v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-appearance-models-for-face-rendering","repo_url":"https://github.com/facebookresearch/multiface","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1808.00362","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}