{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/taming-transformers-for-high-resolution-image","title":"Taming Transformers for High-Resolution Image Synthesis","arxiv_id":"2012.09841","date":"2020-12-17","proceeding":"CVPR 2021 1","authors":["Patrick Esser","Robin Rombach","Björn Ommer"],"abstract":"Designed to learn long-range interactions on sequential data, transformers continue to show state-of-the-art results on a wide variety of tasks. In contrast to CNNs, they contain no inductive bias that prioritizes local interactions. This makes them expressive, but also computationally infeasible for long sequences, such as high-resolution images. We demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers enables them to model and thereby synthesize high-resolution images. We show how to (i) use CNNs to learn a context-rich vocabulary of image constituents, and in turn (ii) utilize transformers to efficiently model their composition within high-resolution images. Our approach is readily applied to conditional synthesis tasks, where both non-spatial information, such as object classes, and spatial information, such as segmentations, can control the generated image. In particular, we present the first results on semantically-guided synthesis of megapixel images with transformers and obtain the state of the art among autoregressive models on class-conditional ImageNet. Code and pretrained models can be found at https://github.com/CompVis/taming-transformers .","url_abs":"https://arxiv.org/abs/2012.09841v3","url_pdf":"https://arxiv.org/pdf/2012.09841v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/CompVis/taming-transformers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/YvanG/VQGAN-CLIP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/dome272/VQGAN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/dome272/vqgan-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/hyn2028/llm-cxr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/joanrod/ocr-vqgan","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/joh-fischer/PlantLDM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/samb-t/unleashing-transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/tgisaturday/taming-transformers-tpu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"unanswered"}},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/v-iashin/SpecVQGAN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/xiaoiker/meta_dpm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/FaceOnLive/DeepFake-Detection-SDK-Linux","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"unanswered"}},{"paper_slug":"taming-transformers-for-high-resolution-image","repo_url":"https://github.com/alibaba/EasyNLP/tree/master/examples/text2image_generation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"deepfake-detection","task_name":"DeepFake Detection"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-outpainting","task_name":"Image Outpainting"},{"task_slug":"image-reconstruction","task_name":"Image Reconstruction"},{"task_slug":"image-to-image-translation","task_name":"Image-to-Image Translation"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"high","task_name":"Vocal Bursts Intensity Prediction"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/deepfake-detection-on-fakeavceleb-1","task":"DeepFake Detection","dataset":"FakeAVCeleb","model":"VQGAN","rank_in_archive_order":9,"of":13,"metrics":{"AP":"55.0","ROC AUC":"51.8"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-celeba-256x256","task":"Image Generation","dataset":"CelebA 256x256","model":"VQGAN","rank_in_archive_order":13,"of":17,"metrics":{"FID":"10.2"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-celeba-hq-256x256","task":"Image Generation","dataset":"CelebA-HQ 256x256","model":"VQGAN+Transformer","rank_in_archive_order":13,"of":19,"metrics":{"FID":"10.2"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-ffhq-256-x-256","task":"Image Generation","dataset":"FFHQ 256 x 256","model":"VQGAN+Transformer","rank_in_archive_order":35,"of":51,"metrics":{"FID":"9.6"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-256x256","task":"Image Generation","dataset":"ImageNet 256x256","model":"VQGAN+Transformer (k=600, p=1.0, a=0.05)","rank_in_archive_order":88,"of":94,"metrics":{"FID":"5.2"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-256x256","task":"Image Generation","dataset":"ImageNet 256x256","model":"VQGAN+Transformer (k=mixed, p=1.0, a=0.005)","rank_in_archive_order":90,"of":94,"metrics":{"FID":"6.59"},"uses_additional_data":false},{"leaderboard":"/sota/image-outpainting-on-lhqc","task":"Image Outpainting","dataset":"LHQC","model":"Taming","rank_in_archive_order":4,"of":4,"metrics":{"Block-FID  (Right Extend)":"22.53","Block-FID (Down Extend)":"26.38","Block-FID (Left Extend)":"-","Block-FID (Up Extend)":"-"},"uses_additional_data":false},{"leaderboard":"/sota/image-reconstruction-on-imagenet","task":"Image Reconstruction","dataset":"ImageNet","model":"Taming-VQGAN (16x16)","rank_in_archive_order":15,"of":15,"metrics":{"FID":"3.64","LPIPS":"0.177","PSNR":"19.93","SSIM":"0.542"},"uses_additional_data":false},{"leaderboard":"/sota/image-reconstruction-on-ultra-high-resolution","task":"Image Reconstruction","dataset":"Ultra-High Resolution Image Reconstruction Benchmark","model":"VQGAN (16x16)","rank_in_archive_order":5,"of":6,"metrics":{"PSNR":"22.91","rFID":"5.95"},"uses_additional_data":false},{"leaderboard":"/sota/image-to-image-translation-on-ade20k-labels","task":"Image-to-Image Translation","dataset":"ADE20K Labels-to-Photos","model":"VQGAN+Transformer","rank_in_archive_order":15,"of":16,"metrics":{"FID":"35.5"},"uses_additional_data":false},{"leaderboard":"/sota/image-to-image-translation-on-coco-stuff","task":"Image-to-Image Translation","dataset":"COCO-Stuff Labels-to-Photos","model":"VQGAN+Transformer","rank_in_archive_order":9,"of":15,"metrics":{"FID":"22.4"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-image-generation-on-conceptual","task":"Text-to-Image Generation","dataset":"Conceptual Captions","model":"VQ-GAN","rank_in_archive_order":5,"of":5,"metrics":{"FID":"28.86"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-image-generation-on-lhqc","task":"Text-to-Image Generation","dataset":"LHQC","model":"Taming","rank_in_archive_order":3,"of":3,"metrics":{"Block-FID":"38.89"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2012.09841","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2012.09841"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/samb-t/unleashing-transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joanrod/ocr-vqgan","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dome272/VQGAN","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xiaoiker/meta_dpm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alibaba/EasyNLP/tree/master/examples/text2image_generation","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YvanG/VQGAN-CLIP","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hyn2028/llm-cxr","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tgisaturday/taming-transformers-tpu","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joh-fischer/PlantLDM","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/v-iashin/SpecVQGAN","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dome272/vqgan-pytorch","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/FaceOnLive/DeepFake-Detection-SDK-Linux","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/CompVis/taming-transformers","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":3,"ran_violates":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"7439a9d4c6be04b0","entry":"adopt_weight","repo":"joanrod/ocr-vqgan","repo_kind":"listed","path":"taming/modules/losses/vqperceptual.py","file_url":"https://github.com/joanrod/ocr-vqgan/blob/HEAD/taming/modules/losses/vqperceptual.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7439a9d4c6be04b0"}},{"code_sha256_prefix":"6ca974beb2276abf","entry":"bchw_to_st","repo":"CompVis/taming-transformers","repo_kind":"official","path":"scripts/sample_conditional.py","file_url":"https://github.com/CompVis/taming-transformers/blob/HEAD/scripts/sample_conditional.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ca974beb2276abf"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"22a83df891d750c6","entry":"hinge_d_loss","repo":"joanrod/ocr-vqgan","repo_kind":"listed","path":"taming/modules/losses/vqperceptual.py","file_url":"https://github.com/joanrod/ocr-vqgan/blob/HEAD/taming/modules/losses/vqperceptual.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"22a83df891d750c6"}},{"code_sha256_prefix":"d2bff574a971c670","entry":"single_image_to_torch","repo":"CompVis/taming-transformers","repo_kind":"official","path":"scripts/sample_conditional.py","file_url":"https://github.com/CompVis/taming-transformers/blob/HEAD/scripts/sample_conditional.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2bff574a971c670"}},{"code_sha256_prefix":"263c54290c553ddd","entry":"vanilla_d_loss","repo":"joanrod/ocr-vqgan","repo_kind":"listed","path":"taming/modules/losses/vqperceptual.py","file_url":"https://github.com/joanrod/ocr-vqgan/blob/HEAD/taming/modules/losses/vqperceptual.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"263c54290c553ddd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}