{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/taming-scalable-visual-tokenizer-for","title":"Taming Scalable Visual Tokenizer for Autoregressive Image Generation","arxiv_id":"2412.02692","date":"2024-12-03","proceeding":null,"authors":["Fengyuan Shi","Zhuoyan Luo","Yixiao Ge","Yujiu Yang","Ying Shan","LiMin Wang"],"abstract":"Existing vector quantization (VQ) methods struggle with scalability, largely attributed to the instability of the codebook that undergoes partial updates during training. The codebook is prone to collapse as utilization decreases, due to the progressively widening distribution gap between non-activated codes and visual features. To solve the problem, we propose Index Backpropagation Quantization (IBQ), a new VQ method for the joint optimization of all codebook embeddings and the visual encoder. Applying a straight-through estimator on the one-hot categorical distribution between the encoded feature and codebook, all codes are differentiable and maintain a consistent latent space with the visual encoder. IBQ enables scalable training of visual tokenizers and, for the first time, achieves a large-scale codebook ($2^{18}$) with high dimension ($256$) and high utilization. Experiments on the standard ImageNet benchmark demonstrate the scalability and superiority of IBQ, achieving competitive results on both reconstruction ($1.00$ rFID) and autoregressive visual generation ($2.05$ gFID). The code and models are available at https://github.com/TencentARC/SEED-Voken.","url_abs":"https://arxiv.org/abs/2412.02692v1","url_pdf":"https://arxiv.org/pdf/2412.02692v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"taming-scalable-visual-tokenizer-for","repo_url":"https://github.com/tencentarc/seed-voken","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"taming-scalable-visual-tokenizer-for","repo_url":"https://github.com/tencentarc/open-magvit2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-reconstruction","task_name":"Image Reconstruction"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-reconstruction-on-imagenet","task":"Image Reconstruction","dataset":"ImageNet","model":"IBQ (16x16)","rank_in_archive_order":6,"of":15,"metrics":{"FID":"1.00","LPIPS":"0.2030"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.02692","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.02692"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tencentarc/seed-voken","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tencentarc/open-magvit2","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":5,"ran_honours":1,"ran_violates":1,"ran":2,"unverified":4},"by_repo_kind":{"listed":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"tencentarc/open-magvit2","repo_kind":"listed","path":"src/Open_MAGVIT2/modules/diffusionmodules/improved_model.py","file_url":"https://github.com/tencentarc/open-magvit2/blob/HEAD/src/Open_MAGVIT2/modules/diffusionmodules/improved_model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"7439a9d4c6be04b0","entry":"adopt_weight","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"7439a9d4c6be04b0"}},{"code_sha256_prefix":"3743889a7a1ed998","entry":"depth_to_space","repo":"tencentarc/open-magvit2","repo_kind":"listed","path":"src/Open_MAGVIT2/modules/diffusionmodules/improved_model.py","file_url":"https://github.com/tencentarc/open-magvit2/blob/HEAD/src/Open_MAGVIT2/modules/diffusionmodules/improved_model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3743889a7a1ed998"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"tencentarc/open-magvit2","repo_kind":"listed","path":"src/Open_MAGVIT2/models/cond_transformer.py","file_url":"https://github.com/tencentarc/open-magvit2/blob/HEAD/src/Open_MAGVIT2/models/cond_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"e70eb36cf08faee4","entry":"get_ckpt_path","repo":"tencentarc/open-magvit2","repo_kind":"listed","path":"src/Open_MAGVIT2/util.py","file_url":"https://github.com/tencentarc/open-magvit2/blob/HEAD/src/Open_MAGVIT2/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e70eb36cf08faee4"}},{"code_sha256_prefix":"22a83df891d750c6","entry":"hinge_d_loss","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"22a83df891d750c6"}},{"code_sha256_prefix":"07d90bd6ae1be7db","entry":"md5_hash","repo":"tencentarc/open-magvit2","repo_kind":"listed","path":"src/Open_MAGVIT2/util.py","file_url":"https://github.com/tencentarc/open-magvit2/blob/HEAD/src/Open_MAGVIT2/util.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"07d90bd6ae1be7db"}},{"code_sha256_prefix":"31543f3e2c18272e","entry":"retrieve","repo":"tencentarc/open-magvit2","repo_kind":"listed","path":"src/Open_MAGVIT2/util.py","file_url":"https://github.com/tencentarc/open-magvit2/blob/HEAD/src/Open_MAGVIT2/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"31543f3e2c18272e"}},{"code_sha256_prefix":"263c54290c553ddd","entry":"vanilla_d_loss","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"263c54290c553ddd"}},{"code_sha256_prefix":"523ecb8d3aa7e677","entry":"depth_to_space3d","repo":"tencentarc/open-magvit2","repo_kind":"listed","path":"src/Open_MAGVIT2/modules/diffusionmodules/improved_video_model.py","file_url":"https://github.com/tencentarc/open-magvit2/blob/HEAD/src/Open_MAGVIT2/modules/diffusionmodules/improved_video_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"523ecb8d3aa7e677"}},{"code_sha256_prefix":"c16bc40c0efe0d76","entry":"flatten_t_dim","repo":"tencentarc/open-magvit2","repo_kind":"listed","path":"src/Open_MAGVIT2/modules/util.py","file_url":"https://github.com/tencentarc/open-magvit2/blob/HEAD/src/Open_MAGVIT2/modules/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c16bc40c0efe0d76"}},{"code_sha256_prefix":"f772835c441ad58e","entry":"preprocess","repo":"tencentarc/open-magvit2","repo_kind":"listed","path":"src/Open_MAGVIT2/modules/util.py","file_url":"https://github.com/tencentarc/open-magvit2/blob/HEAD/src/Open_MAGVIT2/modules/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f772835c441ad58e"}},{"code_sha256_prefix":"4def1646226b2fac","entry":"unflatten_t_dim","repo":"tencentarc/open-magvit2","repo_kind":"listed","path":"src/Open_MAGVIT2/modules/util.py","file_url":"https://github.com/tencentarc/open-magvit2/blob/HEAD/src/Open_MAGVIT2/modules/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4def1646226b2fac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}