Methods › Computer Vision › Likelihood-Based Generative Models › VQ-VAE
VQ-VAE
Introduced by Aaron van den Oord et al. in Neural Discrete Representation Learning
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
VQ-VAE is a type of variational autoencoder that uses vector quantisation to obtain a discrete latent representation. It differs from VAEs in two key ways: the encoder network outputs discrete, rather than continuous, codes; and the prior is learnt rather than static. In order to learn a discrete latent representation, ideas from vector quantisation (VQ) are incorporated. Using the VQ method allows the model to circumvent issues of posterior collapse - where the latents are ignored when they are paired with a powerful autoregressive decoder - typically observed in the VAE framework. Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes.
In Syntology about another paper, arXiv:2606.21970: no code under this exact name (no_code; paper_only). About another paper: this answer is about arXiv:2606.21970, while the archive credits arXiv:1711.00937, so it does not say whether this method has code.
From the lookup's note on this name: The same name also belongs to “Vector Quantised-Variational AutoEncoder (VQ-VAE)”, a method in the paper “Neural Discrete Representation Learning” (arXiv:1711.00937, 7,848 citations), which the lookup finds cited far more than the paper this answer is about (citations not recorded), so this answer may be about a different sense of the name. For arXiv:1711.00937: its page on this site · ask the lookup about it.
Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.
- SAMS-VAE · shares_name_tokens (shares name tokens with this name; a different method) · level V2 · retrieve with
get_reference_implementation(name='SAMS-VAE') - Chroma-VAE · shares_name_tokens (shares name tokens with this name; a different method) · level V2 · retrieve with
get_reference_implementation(name='Chroma-VAE') - Disentangle Variational Autoencoder (D-VAE) · shares_name_tokens (shares name tokens with this name; a different method) · level V2 · retrieve with
get_reference_implementation(name='Disentangle Variational Autoencoder (D-VAE)')
Origin paper of this answer, arXiv:2606.21970 (its page on Syntology).
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
What each level means, quoted from Syntology's server card: V0 “imports”; V1 “executes against a synthesized fixture and returns a value of the recorded type”; V2 “two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs” “neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)”; V3 “paper-derived property tests pass on both of those same two generations.” The card adds: “No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results.” A level quoted in two parts is one definition the card writes as two clauses joined by a double hyphen, each clause quoted on its own (how these levels are quoted).
Papers archive 2025-07-28
archive papers tagged: 197 · with a code link: 82 · where Syntology ran a sample: 35 (32 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (35 of 197 tagged: 32 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 197, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modeling 23 Jun 2025 · 1 repository · arXiv:2506.18680
-
Policy-Based Trajectory Clustering in Offline Reinforcement Learning 10 Jun 2025 · 0 repositories · arXiv:2506.09202
-
STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization 4 Jun 2025 · 1 repository · arXiv:2506.03863
-
VesselGPT: Autoregressive Modeling of Vascular Geometry 19 May 2025 · 1 repository · arXiv:2505.13318
-
M3G: Multi-Granular Gesture Generator for Audio-Driven Full-Body Human Motion Synthesis 13 May 2025 · 0 repositories · arXiv:2505.08293
-
Towards Foundation Models for Experimental Readout Systems Combining Discrete and Continuous Data 13 May 2025 · 1 repository · arXiv:2505.08736
-
Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input 11 Apr 2025 · 0 repositories · arXiv:2504.08449
-
Instruction-Guided Autoregressive Neural Network Parameter Generation 2 Apr 2025 · 0 repositories · arXiv:2504.02012
-
Make Some Noise: Towards LLM audio reasoning and generation using sound tokens 28 Mar 2025 · 0 repositories · arXiv:2503.22275
-
AGIR: Assessing 3D Gait Impairment with Reasoning based on LLMs 23 Mar 2025 · 0 repositories · arXiv:2503.18141
-
Improving Autoregressive Image Generation through Coarse-to-Fine Token Prediction 20 Mar 2025 · 0 repositories · arXiv:2503.16194
-
A Foundation Model for Patient Behavior Monitoring and Suicide Detection 19 Mar 2025 · 0 repositories · arXiv:2503.15221
-
GenM³: Generative Pretrained Multi-path Motion Model for Text Conditional Human Motion Generation 19 Mar 2025 · 0 repositories · arXiv:2503.14919
-
BioSerenity-E1: a self-supervised EEG model for medical applications 13 Mar 2025 · 0 repositories · arXiv:2503.10362
-
UniGenX: Unified Generation of Sequence and Structure with Autoregressive Diffusion 9 Mar 2025 · 0 repositories · arXiv:2503.06687
-
Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cues 5 Mar 2025 · 0 repositories · arXiv:2503.03474
-
CAPS: Context-Aware Priority Sampling for Enhanced Imitation Learning in Autonomous Driving 3 Mar 2025 · 0 repositories · arXiv:2503.01650
-
Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement 11 Feb 2025 · 0 repositories · arXiv:2502.07243
-
Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning 5 Feb 2025 · 0 repositories · arXiv:2502.03275
-
Patch-aware Vector Quantized Codebook Learning for Unsupervised Visual Defect Detection 15 Jan 2025 · 0 repositories · arXiv:2501.09187
-
HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models 1 Jan 2025 · 0 repositories
-
LLM-driven Multimodal and Multi-Identity Listening Head Generation 1 Jan 2025 · 0 repositories
-
TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction 22 Dec 2024 · 0 repositories · arXiv:2412.16919
-
CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls 13 Dec 2024 · 0 repositories · arXiv:2412.09887
-
Adaptive²: Adaptive Domain Mining for Fine-grained Domain Adaptation Modeling 11 Dec 2024 · 0 repositories · arXiv:2412.08198
-
SweetTokenizer: Semantic-Aware Spatial-Temporal Tokenizer for Compact Visual Discretization 11 Dec 2024 · 0 repositories · arXiv:2412.10443
-
VQalAttent: a Transparent Speech Generation Pipeline based on Transformer-learned VQ-VAE Latent Space 22 Nov 2024 · 0 repositories · arXiv:2411.14642
-
Augmenting Training Data with Vector-Quantized Variational Autoencoder for Classifying RF Signals 23 Oct 2024 · 0 repositories · arXiv:2410.18283
-
Gaussian Mixture Vector Quantization with Aggregated Categorical Posterior 14 Oct 2024 · 0 repositories · arXiv:2410.10180
-
InterMask: 3D Human Interaction Generation via Collaborative Masked Modelling 13 Oct 2024 · 1 repository · arXiv:2410.10010
-
Restructuring Vector Quantization with the Rotation Trick 8 Oct 2024 · 3 repositories · arXiv:2410.06424Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 2 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples) · 7 pointer-only (licence)
-
ECHOPulse: ECG controlled echocardio-grams video generation 4 Oct 2024 · 1 repository · arXiv:2410.03143Syntology official (archive's flag): 14 ran · 14 ran (of which 0 constructed an object rather than computing a result; 11 with no instrument failure: 4 honoured, 6 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 14 harvested samples) · 14 pointer-only (licence)
-
COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Models 30 Sep 2024 · 0 repositories · arXiv:2409.20502
-
Comparative Analysis of Pretrained Audio Representations in Music Recommender Systems 13 Sep 2024 · 1 repository · arXiv:2409.08987
-
ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE 12 Sep 2024 · 1 repository · arXiv:2409.07966
-
An End-to-End Approach for Chord-Conditioned Song Generation 10 Sep 2024 · 0 repositories · arXiv:2409.06307
-
Latent 3D Brain MRI Counterfactual 9 Sep 2024 · 0 repositories · arXiv:2409.05585
-
GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer 3 Aug 2024 · 0 repositories · arXiv:2408.01826
-
Synthetic dual image generation for reduction of labeling efforts in semantic segmentation of micrographs with a customized metric function 1 Aug 2024 · 0 repositories · arXiv:2408.00707
-
Quantised Global Autoencoder: A Holistic Approach to Representing Visual Data 16 Jul 2024 · 0 repositories · arXiv:2407.11913
-
Balance of Number of Embedding and their Dimensions in Vector Quantization 6 Jul 2024 · 0 repositories · arXiv:2407.04939
-
From Audio Encoders to Piano Judges: Benchmarking Performance Understanding for Solo Piano 5 Jul 2024 · 0 repositories · arXiv:2407.04518
-
MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers 14 Jun 2024 · 1 repository · arXiv:2406.10163Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation 11 Jun 2024 · 0 repositories · arXiv:2406.07422
-
T2LM: Long-Term 3D Human Motion Generation from Multiple Sentences 2 Jun 2024 · 0 repositories · arXiv:2406.00636
-
Stratified Avatar Generation from Sparse Observations 30 May 2024 · 0 repositories · arXiv:2405.20786
-
Rate-Adaptive Quantization: A Multi-Rate Codebook Adaptation for Vector Quantization-based Generative Models 23 May 2024 · 1 repository · arXiv:2405.14222
-
FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information 21 May 2024 · 1 repository · arXiv:2405.12807
-
Controllable Image Generation With Composed Parallel Token Prediction 10 May 2024 · 0 repositories · arXiv:2405.06535
-
Self-supervised Pre-training of Text Recognizers 1 May 2024 · 1 repository · arXiv:2405.00420
-
MIDGET: Music Conditioned 3D Dance Generation 18 Apr 2024 · 0 repositories · arXiv:2404.12062
-
Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption 17 Apr 2024 · 1 repository · arXiv:2404.11291Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 4 unverified (of 8 harvested samples) · 1 pointer-only (licence)
-
Taming Transformers for Realistic Lidar Point Cloud Generation 8 Apr 2024 · 1 repository · arXiv:2404.05505
-
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction 3 Apr 2024 · 3 repositories · arXiv:2404.02905Syntology community repositories only · 7 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 4 unverified (of 11 harvested samples)
-
A Novel Audio Representation for Music Genre Identification in MIR 1 Apr 2024 · 0 repositories · arXiv:2404.01058
-
ParCo: Part-Coordinating Text-to-Motion Synthesis 27 Mar 2024 · 1 repository · arXiv:2403.18512
-
Contact-aware Human Motion Generation from Textual Descriptions 23 Mar 2024 · 0 repositories · arXiv:2403.15709
-
Dyadic Interaction Modeling for Social Behavior Generation 14 Mar 2024 · 2 repositories · arXiv:2403.09069
-
Vector Quantization for Deep-Learning-Based CSI Feedback in Massive MIMO Systems 12 Mar 2024 · 0 repositories · arXiv:2403.07355
-
Say Anything with Any Style 11 Mar 2024 · 0 repositories · arXiv:2403.06363
-
Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech 26 Feb 2024 · 1 repository · arXiv:2402.16321Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified (of 11 harvested samples) · 11 pointer-only (licence)
-
Neural Language of Thought Models 2 Feb 2024 · 0 repositories · arXiv:2402.01203
-
Dynamic Texture Transfer using PatchMatch and Transformers 1 Feb 2024 · 0 repositories · arXiv:2402.00606
-
HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes 31 Dec 2023 · 0 repositories · arXiv:2401.00365
-
Fast gradient-free activation maximization for neurons in spiking neural networks 28 Dec 2023 · 1 repository · arXiv:2401.10748
-
StyleSpeech: Self-supervised Style Enhancing with VQ-VAE-based Pre-training for Expressive Audiobook Speech Synthesis 19 Dec 2023 · 0 repositories · arXiv:2312.12181
-
T2M-HiFiGPT: Generating High Quality Human Motion from Textual Descriptions with Residual Discrete Representations 17 Dec 2023 · 0 repositories · arXiv:2312.10628
-
GIVT: Generative Infinite-Vocabulary Transformers 4 Dec 2023 · 2 repositories · arXiv:2312.02116
-
SpeechAct: Towards Generating Whole-body Motion from Speech 29 Nov 2023 · 0 repositories · arXiv:2311.17425
-
TLControl: Trajectory and Language Control for Human Motion Synthesis 28 Nov 2023 · 0 repositories · arXiv:2311.17135
-
Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation 27 Nov 2023 · 0 repositories · arXiv:2311.16201
-
Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers 17 Nov 2023 · 0 repositories · arXiv:2311.10242
-
Utilizing VQ-VAE for End-to-End Health Indicator Generation in Predicting Rolling Bearing RUL 17 Nov 2023 · 0 repositories · arXiv:2311.10525
-
scBeacon: single-cell biomarker extraction via identifying paired cell clusters across biological conditions with contrastive siamese networks 5 Nov 2023 · 0 repositories · arXiv:2311.02594
-
Composer Style-specific Symbolic Music Generation Using Vector Quantized Discrete Diffusion Models 21 Oct 2023 · 1 repository · arXiv:2310.14044
-
HumanTOMATO: Text-aligned Whole-body Motion Generation 19 Oct 2023 · 1 repository · arXiv:2310.12978Syntology official (archive's flag): 13 ran · 13 ran (of which 0 constructed an object rather than computing a result; 12 with no instrument failure: 1 honoured, 0 violated, 11 with no contract checked; 1 where Syntology's instrument failed) · 5 unverified (of 18 harvested samples) · 18 pointer-only (licence)
-
Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning 18 Oct 2023 · 0 repositories · arXiv:2310.11731
-
LL-VQ-VAE: Learnable Lattice Vector-Quantization For Efficient Representations 13 Oct 2023 · 0 repositories · arXiv:2310.09382
-
EdVAE: Mitigating Codebook Collapse with Evidential Discrete Variational Autoencoders 9 Oct 2023 · 1 repository · arXiv:2310.05718
-
Finite Scalar Quantization: VQ-VAE Made Simple 27 Sep 2023 · 3 repositories · arXiv:2309.15505Syntology community repositories only · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 2 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 2 pointer-only (licence)
-
Attentive VQ-VAE 20 Sep 2023 · 0 repositories · arXiv:2309.11641
-
AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanism 2 Sep 2023 · 1 repository · arXiv:2309.00796Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Few shot font generation via transferring similarity guided global style and quantization local style 2 Sep 2023 · 1 repository · arXiv:2309.00827Syntology official (archive's flag): 14 ran · 14 ran (of which 9 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 4 where Syntology's instrument failed) · 6 unverified (of 20 harvested samples)
-
QS-TTS: Towards Semi-Supervised Text-to-Speech Synthesis via Vector-Quantized Self-Supervised Speech Representation Learning 31 Aug 2023 · 1 repository · arXiv:2309.00126
-
Priority-Centric Human Motion Generation in Discrete Latent Space 28 Aug 2023 · 0 repositories · arXiv:2308.14480
-
Burnt area extraction from high-resolution satellite images based on anomaly detection 25 Aug 2023 · 0 repositories · arXiv:2308.13367
-
Can Language Models Learn to Listen? 21 Aug 2023 · 0 repositories · arXiv:2308.10897
-
Neural Categorical Priors for Physics-Based Character Control 14 Aug 2023 · 0 repositories · arXiv:2308.07200
-
Controlling Character Motions without Observable Driving Source 11 Aug 2023 · 0 repositories · arXiv:2308.06025
-
MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion 8 Aug 2023 · 1 repository · arXiv:2308.04249Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)
-
VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs 4 Aug 2023 · 1 repository · arXiv:2308.02117Syntology official (archive's flag): 23 ran · 23 ran (of which 1 constructed an object rather than computing a result; 10 with no instrument failure: 1 honoured, 4 violated, 5 with no contract checked; 13 where Syntology's instrument failed) · 7 unverified (of 30 harvested samples) · 30 pointer-only (licence)
-
Online Clustered Codebook 27 Jul 2023 · 1 repository · arXiv:2307.15139Syntology official (archive's flag): 9 ran · 9 ran (of which 2 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 1 violated, 7 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 10 harvested samples) · 2 pointer-only (licence)
-
Tokenization with Factorized Subword Encoding 13 Jun 2023 · 1 repository · arXiv:2306.07764
-
DualVAE: Controlling Colours of Generated and Real Images 30 May 2023 · 0 repositories · arXiv:2305.18769
-
Visual Affordance Prediction for Guiding Robot Exploration 28 May 2023 · 0 repositories · arXiv:2305.17783
-
A One-Class Classifier for the Detection of GAN Manipulated Multi-Spectral Satellite Images 19 May 2023 · 0 repositories · arXiv:2305.11795
-
QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation 18 May 2023 · 1 repository · arXiv:2305.11094Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 5 pointer-only (licence)
-
Catch Missing Details: Image Reconstruction with Frequency Augmented Variational Autoencoder 4 May 2023 · 1 repository · arXiv:2305.02541Syntology official (archive's flag): 13 ran · 13 ran (of which 0 constructed an object rather than computing a result; 11 with no instrument failure: 0 honoured, 3 violated, 8 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 15 harvested samples) · 3 pointer-only (licence)
-
TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integration 5 Apr 2023 · 1 repository · arXiv:2304.02419Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)
-
MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion 24 Mar 2023 · 0 repositories · arXiv:2303.14139
Tasks archive 2025-07-28
20 shown of 171 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections