{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/givt-generative-infinite-vocabulary","title":"GIVT: Generative Infinite-Vocabulary Transformers","arxiv_id":"2312.02116","date":"2023-12-04","proceeding":null,"authors":["Michael Tschannen","Cian Eastwood","Fabian Mentzer"],"abstract":"We introduce Generative Infinite-Vocabulary Transformers (GIVT) which generate vector sequences with real-valued entries, instead of discrete tokens from a finite vocabulary. To this end, we propose two surprisingly simple modifications to decoder-only transformers: 1) at the input, we replace the finite-vocabulary lookup table with a linear projection of the input vectors; and 2) at the output, we replace the logits prediction (usually mapped to a categorical distribution) with the parameters of a multivariate Gaussian mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT, where transformers are used to model the discrete latent sequences of a VQ-VAE, we use GIVT to model the unquantized real-valued latent sequences of a $\\beta$-VAE. In class-conditional image generation GIVT outperforms VQ-GAN (and improved variants thereof) as well as MaskGIT, and achieves performance competitive with recent latent diffusion models. Finally, we obtain strong results outside of image generation when applying GIVT to panoptic segmentation and depth estimation with a VAE variant of the UViM framework.","url_abs":"https://arxiv.org/abs/2312.02116v4","url_pdf":"https://arxiv.org/pdf/2312.02116v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"givt-generative-infinite-vocabulary","repo_url":"https://github.com/google-research/big_vision","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"givt-generative-infinite-vocabulary","repo_url":"https://github.com/shaochenze/ear","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"conditional-image-generation","task_name":"Conditional Image Generation"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"panoptic-segmentation","task_name":"Panoptic Segmentation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"vq-vae","method_name":"VQ-VAE"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-generation-on-imagenet-256x256","task":"Image Generation","dataset":"ImageNet 256x256","model":"GIVT-Causal-L+A","rank_in_archive_order":61,"of":94,"metrics":{"FID":"2.59"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-512x512","task":"Image Generation","dataset":"ImageNet 512x512","model":"GIVT-Causal-L+A","rank_in_archive_order":40,"of":52,"metrics":{"FID":"2.92"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.02116","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}