{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/online-clustered-codebook","title":"Online Clustered Codebook","arxiv_id":"2307.15139","date":"2023-07-27","proceeding":"ICCV 2023 1","authors":["Chuanxia Zheng","Andrea Vedaldi"],"abstract":"Vector Quantisation (VQ) is experiencing a comeback in machine learning, where it is increasingly used in representation learning. However, optimizing the codevectors in existing VQ-VAE is not entirely trivial. A problem is codebook collapse, where only a small subset of codevectors receive gradients useful for their optimisation, whereas a majority of them simply ``dies off'' and is never updated or used. This limits the effectiveness of VQ for learning larger codebooks in complex computer vision tasks that require high-capacity representations. In this paper, we present a simple alternative method for online codebook learning, Clustering VQ-VAE (CVQ-VAE). Our approach selects encoded features as anchors to update the ``dead'' codevectors, while optimising the codebooks which are alive via the original loss. This strategy brings unused codevectors closer in distribution to the encoded features, increasing the likelihood of being chosen and optimized. We extensively validate the generalization capability of our quantiser on various datasets, tasks (e.g. reconstruction and generation), and architectures (e.g. VQ-VAE, VQGAN, LDM). Our CVQ-VAE can be easily integrated into the existing models with just a few lines of code.","url_abs":"https://arxiv.org/abs/2307.15139v1","url_pdf":"https://arxiv.org/pdf/2307.15139v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"online-clustered-codebook","repo_url":"https://github.com/lyndonzheng/cvq-vae","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"vq-vae","method_name":"VQ-VAE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2307.15139","atlas_url":"https://app.syntology.ai/?focus=2307.15139","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.15139"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lyndonzheng/cvq-vae","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/lyndonzheng/CVQ-VAE","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"ran_fixture":1,"ran_violates":1,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"16bbb393dd49558f","entry":"FeaturePool","repo":"lyndonzheng/CVQ-VAE","repo_kind":"official","path":"quantise.py","file_url":"https://github.com/lyndonzheng/CVQ-VAE/blob/HEAD/quantise.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16bbb393dd49558f"}},{"code_sha256_prefix":"421dff121c22ecee","entry":"VectorQuantiser","repo":"lyndonzheng/CVQ-VAE","repo_kind":"official","path":"quantise.py","file_url":"https://github.com/lyndonzheng/CVQ-VAE/blob/HEAD/quantise.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"421dff121c22ecee"}},{"code_sha256_prefix":"4cee7c1861294d59","entry":"calculate_frechet_distance","repo":"lyndonzheng/cvq-vae","repo_kind":"official","path":"fid_score.py","file_url":"https://github.com/lyndonzheng/cvq-vae/blob/HEAD/fid_score.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cee7c1861294d59"}},{"code_sha256_prefix":"e501824778deaaba","entry":"imread","repo":"lyndonzheng/cvq-vae","repo_kind":"official","path":"fid_score.py","file_url":"https://github.com/lyndonzheng/cvq-vae/blob/HEAD/fid_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e501824778deaaba"}},{"code_sha256_prefix":"ab4109634b75ef8b","entry":"is_image_file","repo":"lyndonzheng/cvq-vae","repo_kind":"official","path":"image_folder.py","file_url":"https://github.com/lyndonzheng/cvq-vae/blob/HEAD/image_folder.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab4109634b75ef8b"}},{"code_sha256_prefix":"2ada525b8065407b","entry":"make_dataset","repo":"lyndonzheng/cvq-vae","repo_kind":"official","path":"image_folder.py","file_url":"https://github.com/lyndonzheng/cvq-vae/blob/HEAD/image_folder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ada525b8065407b"}},{"code_sha256_prefix":"f2c64c9aef6136db","entry":"make_dataset_txt","repo":"lyndonzheng/cvq-vae","repo_kind":"official","path":"image_folder.py","file_url":"https://github.com/lyndonzheng/cvq-vae/blob/HEAD/image_folder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2c64c9aef6136db"}},{"code_sha256_prefix":"0d6fa4426be1d4f0","entry":"tensor2array","repo":"lyndonzheng/cvq-vae","repo_kind":"official","path":"util.py","file_url":"https://github.com/lyndonzheng/cvq-vae/blob/HEAD/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d6fa4426be1d4f0"}},{"code_sha256_prefix":"ac25d7d08e59e05c","entry":"tensor2im","repo":"lyndonzheng/cvq-vae","repo_kind":"official","path":"util.py","file_url":"https://github.com/lyndonzheng/cvq-vae/blob/HEAD/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac25d7d08e59e05c"}},{"code_sha256_prefix":"3c77e5bea8e44a31","entry":"get_activations","repo":"lyndonzheng/cvq-vae","repo_kind":"official","path":"fid_score.py","file_url":"https://github.com/lyndonzheng/cvq-vae/blob/HEAD/fid_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c77e5bea8e44a31"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}