{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/restructuring-vector-quantization-with-the","title":"Restructuring Vector Quantization with the Rotation Trick","arxiv_id":"2410.06424","date":"2024-10-08","proceeding":null,"authors":["Christopher Fifty","Ronald G. Junkins","Dennis Duan","Aniketh Iger","Jerry W. Liu","Ehsan Amid","Sebastian Thrun","Christopher Ré"],"abstract":"Vector Quantized Variational AutoEncoders (VQ-VAEs) are designed to compress a continuous input to a discrete latent space and reconstruct it with minimal distortion. They operate by maintaining a set of vectors -- often referred to as the codebook -- and quantizing each encoder output to the nearest vector in the codebook. However, as vector quantization is non-differentiable, the gradient to the encoder flows around the vector quantization layer rather than through it in a straight-through approximation. This approximation may be undesirable as all information from the vector quantization operation is lost. In this work, we propose a way to propagate gradients through the vector quantization layer of VQ-VAEs. We smoothly transform each encoder output into its corresponding codebook vector via a rotation and rescaling linear transformation that is treated as a constant during backpropagation. As a result, the relative magnitude and angle between encoder output and codebook vector becomes encoded into the gradient as it propagates through the vector quantization layer and back to the encoder. Across 11 different VQ-VAE training paradigms, we find this restructuring improves reconstruction metrics, codebook utilization, and quantization error. Our code is available at https://github.com/cfifty/rotation_trick.","url_abs":"https://arxiv.org/abs/2410.06424v1","url_pdf":"https://arxiv.org/pdf/2410.06424v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"restructuring-vector-quantization-with-the","repo_url":"https://github.com/cfifty/rotation_trick","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"restructuring-vector-quantization-with-the","repo_url":"https://github.com/lucidrains/vector-quantize-pytorch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"restructuring-vector-quantization-with-the","repo_url":"https://github.com/EdoardoBotta/RQ-VAE-Recommender","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"set","method_name":"SET"},{"method_slug":"vq-vae","method_name":"VQ-VAE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.06424","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.06424"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EdoardoBotta/RQ-VAE-Recommender","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cfifty/rotation_trick","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/vector-quantize-pytorch","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":2,"ran_violates":2,"ran_honours":1},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"c3a6b977022957cb","entry":"Normalize","repo":"cfifty/rotation_trick","repo_kind":"official","path":"src/modules/blocks.py","file_url":"https://github.com/cfifty/rotation_trick/blob/HEAD/src/modules/blocks.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3a6b977022957cb"}},{"code_sha256_prefix":"3d1b48b9b7c6afe0","entry":"data_to_image","repo":"cfifty/rotation_trick","repo_kind":"official","path":"src/reconstruction_fid.py","file_url":"https://github.com/cfifty/rotation_trick/blob/HEAD/src/reconstruction_fid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d1b48b9b7c6afe0"}},{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":"cfifty/rotation_trick","repo_kind":"official","path":"src/local_vector_quantize_pytorch/vector_quantize_pytorch.py","file_url":"https://github.com/cfifty/rotation_trick/blob/HEAD/src/local_vector_quantize_pytorch/vector_quantize_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"cfifty/rotation_trick","repo_kind":"official","path":"src/local_vector_quantize_pytorch/vector_quantize_pytorch.py","file_url":"https://github.com/cfifty/rotation_trick/blob/HEAD/src/local_vector_quantize_pytorch/vector_quantize_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"13e83475b653975d","entry":"get_model_params","repo":"cfifty/rotation_trick","repo_kind":"official","path":"src/models/model_utils.py","file_url":"https://github.com/cfifty/rotation_trick/blob/HEAD/src/models/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"13e83475b653975d"}},{"code_sha256_prefix":"f3232418205f7cbd","entry":"identity","repo":"cfifty/rotation_trick","repo_kind":"official","path":"src/local_vector_quantize_pytorch/vector_quantize_pytorch.py","file_url":"https://github.com/cfifty/rotation_trick/blob/HEAD/src/local_vector_quantize_pytorch/vector_quantize_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3232418205f7cbd"}},{"code_sha256_prefix":"3137073275f8c21a","entry":"nonlinearity","repo":"cfifty/rotation_trick","repo_kind":"official","path":"src/modules/blocks.py","file_url":"https://github.com/cfifty/rotation_trick/blob/HEAD/src/modules/blocks.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3137073275f8c21a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}