{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/robust-training-of-vector-quantized","title":"Robust Training of Vector Quantized Bottleneck Models","arxiv_id":"2005.08520","date":"2020-05-18","proceeding":null,"authors":["Adrian Łańcucki","Jan Chorowski","Guillaume Sanchez","Ricard Marxer","Nanxin Chen","Hans J. G. A. Dolfing","Sameer Khurana","Tanel Alumäe","Antoine Laurent"],"abstract":"In this paper we demonstrate methods for reliable and efficient training of discrete representation using Vector-Quantized Variational Auto-Encoder models (VQ-VAEs). Discrete latent variable models have been shown to learn nontrivial representations of speech, applicable to unsupervised voice conversion and reaching state-of-the-art performance on unit discovery tasks. For unsupervised representation learning, they became viable alternatives to continuous latent variable models such as the Variational Auto-Encoder (VAE). However, training deep discrete variable models is challenging, due to the inherent non-differentiability of the discretization operation. In this paper we focus on VQ-VAE, a state-of-the-art discrete bottleneck model shown to perform on par with its continuous counterparts. It quantizes encoder outputs with on-line $k$-means clustering. We show that the codebook learning can suffer from poor initialization and non-stationarity of clustered encoder outputs. We demonstrate that these can be successfully overcome by increasing the learning rate for the codebook and periodic date-dependent codeword re-initialization. As a result, we achieve more robust training across different tasks, and significantly increase the usage of latent codewords even for large codebooks. This has practical benefit, for instance, in unsupervised representation learning, where large codebooks may lead to disentanglement of latent representations.","url_abs":"https://arxiv.org/abs/2005.08520v1","url_pdf":"https://arxiv.org/pdf/2005.08520v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"robust-training-of-vector-quantized","repo_url":"https://github.com/distsup/DistSup","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"voice-conversion","task_name":"Voice Conversion"}],"methods":[{"method_slug":"vq-vae","method_name":"VQ-VAE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2005.08520","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.08520"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/distsup/DistSup","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fa416c80efcfc519","entry":"flatten_grid_locations","repo":"distsup/DistSup","repo_kind":"official","path":"distsup/augment.py","file_url":"https://github.com/distsup/DistSup/blob/HEAD/distsup/augment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa416c80efcfc519"}},{"code_sha256_prefix":"6e0b3f5632e9d409","entry":"get_grid_locations","repo":"distsup/DistSup","repo_kind":"official","path":"distsup/augment.py","file_url":"https://github.com/distsup/DistSup/blob/HEAD/distsup/augment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e0b3f5632e9d409"}},{"code_sha256_prefix":"054f37d3abef18f0","entry":"get_partial_sampler","repo":"distsup/DistSup","repo_kind":"official","path":"distsup/data.py","file_url":"https://github.com/distsup/DistSup/blob/HEAD/distsup/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"054f37d3abef18f0"}},{"code_sha256_prefix":"924130f28599814b","entry":"glob_latest","repo":"distsup/DistSup","repo_kind":"official","path":"distsup/checkpoints.py","file_url":"https://github.com/distsup/DistSup/blob/HEAD/distsup/checkpoints.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"924130f28599814b"}},{"code_sha256_prefix":"372e1fcc9c84f60d","entry":"latest_checkpoint","repo":"distsup/DistSup","repo_kind":"official","path":"distsup/checkpoints.py","file_url":"https://github.com/distsup/DistSup/blob/HEAD/distsup/checkpoints.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"372e1fcc9c84f60d"}},{"code_sha256_prefix":"5de636b89e261eb5","entry":"read_scoring_map","repo":"distsup/DistSup","repo_kind":"official","path":"distsup/models/distribution_matching.py","file_url":"https://github.com/distsup/DistSup/blob/HEAD/distsup/models/distribution_matching.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5de636b89e261eb5"}},{"code_sha256_prefix":"c607092572584496","entry":"sparse_image_warp","repo":"distsup/DistSup","repo_kind":"official","path":"distsup/augment.py","file_url":"https://github.com/distsup/DistSup/blob/HEAD/distsup/augment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c607092572584496"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}