{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deferred-neural-rendering-image-synthesis","title":"Deferred Neural Rendering: Image Synthesis using Neural Textures","arxiv_id":"1904.12356","date":"2019-04-28","proceeding":null,"authors":["Justus Thies","Michael Zollhöfer","Matthias Nießner"],"abstract":"The modern computer graphics pipeline can synthesize images at remarkable\nvisual quality; however, it requires well-defined, high-quality 3D content as\ninput. In this work, we explore the use of imperfect 3D content, for instance,\nobtained from photo-metric reconstructions with noisy and incomplete surface\ngeometry, while still aiming to produce photo-realistic (re-)renderings. To\naddress this challenging problem, we introduce Deferred Neural Rendering, a new\nparadigm for image synthesis that combines the traditional graphics pipeline\nwith learnable components. Specifically, we propose Neural Textures, which are\nlearned feature maps that are trained as part of the scene capture process.\nSimilar to traditional textures, neural textures are stored as maps on top of\n3D mesh proxies; however, the high-dimensional feature maps contain\nsignificantly more information, which can be interpreted by our new deferred\nneural rendering pipeline. Both neural textures and deferred neural renderer\nare trained end-to-end, enabling us to synthesize photo-realistic images even\nwhen the original 3D content was imperfect. In contrast to traditional,\nblack-box 2D generative neural networks, our 3D representation gives us\nexplicit control over the generated output, and allows for a wide range of\napplication domains. For instance, we can synthesize temporally-consistent\nvideo re-renderings of recorded 3D scenes as our representation is inherently\nembedded in 3D space. This way, neural textures can be utilized to coherently\nre-render or manipulate existing video content in both static and dynamic\nenvironments at real-time rates. We show the effectiveness of our approach in\nseveral experiments on novel view synthesis, scene editing, and facial\nreenactment, and compare to state-of-the-art approaches that leverage the\nstandard graphics pipeline as well as conventional generative neural networks.","url_abs":"http://arxiv.org/abs/1904.12356v1","url_pdf":"http://arxiv.org/pdf/1904.12356v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deferred-neural-rendering-image-synthesis","repo_url":"https://github.com/Leminhbinh0209/AAAI22-ADD","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"deferred-neural-rendering-image-synthesis","repo_url":"https://github.com/SSRSGJYD/NeuralTexture","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"deferred-neural-rendering-image-synthesis","repo_url":"https://github.com/flynn-chen/faceforensics_benchmark","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"deferred-neural-rendering-image-synthesis","repo_url":"https://github.com/leminhbinh0209/add","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"neural-rendering","task_name":"Neural Rendering"},{"task_slug":"novel-view-synthesis","task_name":"Novel View Synthesis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1904.12356","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}