{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unified-generative-adversarial-networks-for","title":"Unified Generative Adversarial Networks for Controllable Image-to-Image Translation","arxiv_id":"1912.06112","date":"2019-12-12","proceeding":null,"authors":["Hao Tang","Hong Liu","Nicu Sebe"],"abstract":"We propose a unified Generative Adversarial Network (GAN) for controllable image-to-image translation, i.e., transferring an image from a source to a target domain guided by controllable structures. In addition to conditioning on a reference image, we show how the model can generate images conditioned on controllable structures, e.g., class labels, object keypoints, human skeletons, and scene semantic maps. The proposed model consists of a single generator and a discriminator taking a conditional image and the target controllable structure as input. In this way, the conditional image can provide appearance information and the controllable structure can provide the structure information for generating the target result. Moreover, our model learns the image-to-image mapping through three novel losses, i.e., color loss, controllable structure guided cycle-consistency loss, and controllable structure guided self-content preserving loss. Also, we present the Fr\\'echet ResNet Distance (FRD) to evaluate the quality of the generated images. Experiments on two challenging image translation tasks, i.e., hand gesture-to-gesture translation and cross-view image translation, show that our model generates convincing results, and significantly outperforms other state-of-the-art methods on both tasks. Meanwhile, the proposed framework is a unified solution, thus it can be applied to solving other controllable structure guided image translation tasks such as landmark guided facial expression translation and keypoint guided person image generation. To the best of our knowledge, we are the first to make one GAN framework work on all such controllable structure guided image translation tasks. Code is available at https://github.com/Ha0Tang/GestureGAN.","url_abs":"https://arxiv.org/abs/1912.06112v2","url_pdf":"https://arxiv.org/pdf/1912.06112v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unified-generative-adversarial-networks-for","repo_url":"https://github.com/Ha0Tang/GestureGAN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"facial-expression-translation","task_name":"Facial Expression Translation"},{"task_slug":null,"task_name":"Generative Adversarial Network"},{"task_slug":"gesture-to-gesture-translation","task_name":"Gesture-to-Gesture Translation"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-to-image-translation","task_name":"Image-to-Image Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-view-image-to-image-translation-on-2","task":"Cross-View Image-to-Image Translation","dataset":"Dayton (256×256) - aerial-to-ground","model":"UniGAN","rank_in_archive_order":6,"of":6,"metrics":{"KL":"5.17","PSNR":"22.0273","SD":"17.6542","SSIM":"0.3357"},"uses_additional_data":false},{"leaderboard":"/sota/cross-view-image-to-image-translation-on-1","task":"Cross-View Image-to-Image Translation","dataset":"Dayton (64x64) - ground-to-aerial","model":"UniGAN","rank_in_archive_order":5,"of":5,"metrics":{"LPIPS":"0.4527"},"uses_additional_data":false},{"leaderboard":"/sota/cross-view-image-to-image-translation-on","task":"Cross-View Image-to-Image Translation","dataset":"Dayton (64×64) - aerial-to-ground","model":"UniGAN","rank_in_archive_order":3,"of":5,"metrics":{"KL":"2.16","LPIPS":"0.3817","PSNR":"23.3632","SD":"16.4788","SSIM":"0.5064"},"uses_additional_data":false},{"leaderboard":"/sota/cross-view-image-to-image-translation-on-4","task":"Cross-View Image-to-Image Translation","dataset":"cvusa","model":"UniGAN","rank_in_archive_order":1,"of":7,"metrics":{"KL":"2.6","PSNR":"22.8223","SD":"19.8276","SSIM":"0.5366"},"uses_additional_data":false},{"leaderboard":"/sota/gesture-to-gesture-translation-on-ntu-hand","task":"Gesture-to-Gesture Translation","dataset":"NTU Hand Digit","model":"UniGAN","rank_in_archive_order":6,"of":6,"metrics":{"AMT":"29.3","FID":"6.7493","FRD":"1.7401","IS":"2.3783","PSNR":"32.6574"},"uses_additional_data":false},{"leaderboard":"/sota/gesture-to-gesture-translation-on-senz3d","task":"Gesture-to-Gesture Translation","dataset":"Senz3D","model":"UniGAN","rank_in_archive_order":6,"of":6,"metrics":{"AMT":"27.6","FID":"12.4465","FRD":"2.2104","IS":"2.2159","PSNR":"31.542"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1912.06112","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}