{"url":"/method/vq-vae","slug":"vq-vae","name":"VQ-VAE","full_name":"VQ-VAE","full_name_withheld":false,"description_markdown":"**VQ-VAE** is a type of variational autoencoder that uses vector quantisation to obtain a discrete latent representation. It differs from [VAEs](https://paperswithcode.com/method/vae) in two key ways: the encoder network outputs discrete, rather than continuous, codes; and the prior is learnt rather than static. In order to learn a discrete latent representation, ideas from vector quantisation (VQ) are incorporated. Using the VQ method allows the model to circumvent issues of posterior collapse - where the latents are ignored when they are paired with a powerful autoregressive decoder - typically observed in the VAE framework. Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Neural Discrete Representation Learning","paper":"/paper/neural-discrete-representation-learning","first_author":"Aaron van den Oord","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/neural-discrete-representation-learning"},"source":{"url":"http://arxiv.org/abs/1711.00937v2","title":"Neural Discrete Representation Learning","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Computer Vision","area_id":"computer-vision","collection":"Likelihood-Based Generative Models","url":"/methods/category/likelihood-based-generative-models","pwc_aliases":[]},{"area":"Computer Vision","area_id":"computer-vision","collection":"Generative Models","url":"/methods/category/generative-models","pwc_aliases":[]}],"n_papers_tagged":197,"archive_num_papers":197,"papers_newest_first":[{"paper":"/paper/duetgen-music-driven-two-person-dance","title":"DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modeling","date":"2025-06-23","arxiv_id":"2506.18680","n_code_links":1,"syntology":null},{"paper":null,"title":"Policy-Based Trajectory Clustering in Offline Reinforcement Learning","date":"2025-06-10","arxiv_id":"2506.09202","n_code_links":0,"syntology":null},{"paper":"/paper/star-learning-diverse-robot-skill","title":"STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization","date":"2025-06-04","arxiv_id":"2506.03863","n_code_links":1,"syntology":null},{"paper":"/paper/vesselgpt-autoregressive-modeling-of-vascular","title":"VesselGPT: Autoregressive Modeling of Vascular Geometry","date":"2025-05-19","arxiv_id":"2505.13318","n_code_links":1,"syntology":null},{"paper":null,"title":"M3G: Multi-Granular Gesture Generator for Audio-Driven Full-Body Human Motion Synthesis","date":"2025-05-13","arxiv_id":"2505.08293","n_code_links":0,"syntology":null},{"paper":"/paper/towards-foundation-models-for-experimental","title":"Towards Foundation Models for Experimental Readout Systems Combining Discrete and Continuous Data","date":"2025-05-13","arxiv_id":"2505.08736","n_code_links":1,"syntology":null},{"paper":null,"title":"Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input","date":"2025-04-11","arxiv_id":"2504.08449","n_code_links":0,"syntology":null},{"paper":null,"title":"Instruction-Guided Autoregressive Neural Network Parameter Generation","date":"2025-04-02","arxiv_id":"2504.02012","n_code_links":0,"syntology":null},{"paper":null,"title":"Make Some Noise: Towards LLM audio reasoning and generation using sound tokens","date":"2025-03-28","arxiv_id":"2503.22275","n_code_links":0,"syntology":null},{"paper":null,"title":"AGIR: Assessing 3D Gait Impairment with Reasoning based on LLMs","date":"2025-03-23","arxiv_id":"2503.18141","n_code_links":0,"syntology":null},{"paper":null,"title":"Improving Autoregressive Image Generation through Coarse-to-Fine Token Prediction","date":"2025-03-20","arxiv_id":"2503.16194","n_code_links":0,"syntology":null},{"paper":null,"title":"A Foundation Model for Patient Behavior Monitoring and Suicide Detection","date":"2025-03-19","arxiv_id":"2503.15221","n_code_links":0,"syntology":null},{"paper":null,"title":"GenM$^3$: Generative Pretrained Multi-path Motion Model for Text Conditional Human Motion Generation","date":"2025-03-19","arxiv_id":"2503.14919","n_code_links":0,"syntology":null},{"paper":null,"title":"BioSerenity-E1: a self-supervised EEG model for medical applications","date":"2025-03-13","arxiv_id":"2503.10362","n_code_links":0,"syntology":null},{"paper":null,"title":"UniGenX: Unified Generation of Sequence and Structure with Autoregressive Diffusion","date":"2025-03-09","arxiv_id":"2503.06687","n_code_links":0,"syntology":null},{"paper":null,"title":"Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cues","date":"2025-03-05","arxiv_id":"2503.03474","n_code_links":0,"syntology":null},{"paper":null,"title":"CAPS: Context-Aware Priority Sampling for Enhanced Imitation Learning in Autonomous Driving","date":"2025-03-03","arxiv_id":"2503.01650","n_code_links":0,"syntology":null},{"paper":null,"title":"Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement","date":"2025-02-11","arxiv_id":"2502.07243","n_code_links":0,"syntology":null},{"paper":null,"title":"Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning","date":"2025-02-05","arxiv_id":"2502.03275","n_code_links":0,"syntology":null},{"paper":null,"title":"Patch-aware Vector Quantized Codebook Learning for Unsupervised Visual Defect Detection","date":"2025-01-15","arxiv_id":"2501.09187","n_code_links":0,"syntology":null},{"paper":null,"title":"HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models","date":"2025-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"LLM-driven Multimodal and Multi-Identity Listening Head Generation","date":"2025-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction","date":"2024-12-22","arxiv_id":"2412.16919","n_code_links":0,"syntology":null},{"paper":null,"title":"CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls","date":"2024-12-13","arxiv_id":"2412.09887","n_code_links":0,"syntology":null},{"paper":null,"title":"Adaptive$^2$: Adaptive Domain Mining for Fine-grained Domain Adaptation Modeling","date":"2024-12-11","arxiv_id":"2412.08198","n_code_links":0,"syntology":null},{"paper":null,"title":"SweetTokenizer: Semantic-Aware Spatial-Temporal Tokenizer for Compact Visual Discretization","date":"2024-12-11","arxiv_id":"2412.10443","n_code_links":0,"syntology":null},{"paper":null,"title":"VQalAttent: a Transparent Speech Generation Pipeline based on Transformer-learned VQ-VAE Latent Space","date":"2024-11-22","arxiv_id":"2411.14642","n_code_links":0,"syntology":null},{"paper":null,"title":"Augmenting Training Data with Vector-Quantized Variational Autoencoder for Classifying RF Signals","date":"2024-10-23","arxiv_id":"2410.18283","n_code_links":0,"syntology":null},{"paper":null,"title":"Gaussian Mixture Vector Quantization with Aggregated Categorical Posterior","date":"2024-10-14","arxiv_id":"2410.10180","n_code_links":0,"syntology":null},{"paper":"/paper/intermask-3d-human-interaction-generation-via","title":"InterMask: 3D Human Interaction Generation via Collaborative Masked Modelling","date":"2024-10-13","arxiv_id":"2410.10010","n_code_links":1,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/quantization","name":"Quantization","papers":30},{"task":"/task/decoder","name":"Decoder","papers":28},{"task":"/task/image-generation","name":"Image Generation","papers":23},{"task":"/task/representation-learning","name":"Representation Learning","papers":17},{"task":"/task/motion-generation","name":"Motion Generation","papers":13},{"task":"/task/language-modelling","name":"Language Modelling","papers":12},{"task":"/task/motion-synthesis","name":"Motion Synthesis","papers":11},{"task":"/task/image-reconstruction","name":"Image Reconstruction","papers":10},{"task":"/task/language-modeling","name":"Language Modeling","papers":10},{"task":"/task/speech-synthesis","name":"Speech Synthesis","papers":10},{"task":"/task/disentanglement","name":"Disentanglement","papers":7},{"task":"/task/video-generation","name":"Video Generation","papers":7},{"task":"/task/clustering","name":"Clustering","papers":6},{"task":"/task/denoising","name":"Denoising","papers":6},{"task":"/task/diversity","name":"Diversity","papers":6},{"task":"/task/music-generation","name":"Music Generation","papers":6},{"task":"/task/text-to-image-generation","name":"Text-to-Image Generation","papers":6},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":6},{"task":"/task/large-language-model","name":"Large Language Model","papers":5},{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":5}],"tasks_shown":20,"n_tasks":171,"usage_by_year":[{"year":"2017","papers":1},{"year":"2018","papers":2},{"year":"2019","papers":6},{"year":"2020","papers":19},{"year":"2021","papers":29},{"year":"2022","papers":31},{"year":"2023","papers":46},{"year":"2024","papers":41},{"year":"2025","papers":22}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/vq-vae"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}