{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/not-all-image-regions-matter-masked-vector-1","title":"Not All Image Regions Matter: Masked Vector Quantization for Autoregressive Image Generation","arxiv_id":"2305.13607","date":"2023-05-23","proceeding":"CVPR 2023 1","authors":["Mengqi Huang","Zhendong Mao","Quan Wang","Yongdong Zhang"],"abstract":"Existing autoregressive models follow the two-stage generation paradigm that first learns a codebook in the latent space for image reconstruction and then completes the image generation autoregressively based on the learned codebook. However, existing codebook learning simply models all local region information of images without distinguishing their different perceptual importance, which brings redundancy in the learned codebook that not only limits the next stage's autoregressive model's ability to model important structure but also results in high training cost and slow generation speed. In this study, we borrow the idea of importance perception from classical image coding theory and propose a novel two-stage framework, which consists of Masked Quantization VAE (MQ-VAE) and Stackformer, to relieve the model from modeling redundancy. Specifically, MQ-VAE incorporates an adaptive mask module for masking redundant region features before quantization and an adaptive de-mask module for recovering the original grid image feature map to faithfully reconstruct the original images after quantization. Then, Stackformer learns to predict the combination of the next code and its position in the feature map. Comprehensive experiments on various image generation validate our effectiveness and efficiency. Code will be released at https://github.com/CrossmodalGroup/MaskedVectorQuantization.","url_abs":"https://arxiv.org/abs/2305.13607v1","url_pdf":"https://arxiv.org/pdf/2305.13607v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"not-all-image-regions-matter-masked-vector-1","repo_url":"https://github.com/crossmodalgroup/maskedvectorquantization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-reconstruction","task_name":"Image Reconstruction"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2305.13607","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.13607"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/crossmodalgroup/maskedvectorquantization","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":4,"ran_draft_wrong":6,"ran_honours":1,"ran_fixture":1,"ran":1,"unverified":5},"by_repo_kind":{"official":{"samples":18,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7c49b92babb27a7d","entry":"default","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/discriminator/stylegan_lucidrains.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/discriminator/stylegan_lucidrains.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c49b92babb27a7d"}},{"code_sha256_prefix":"c3a6b977022957cb","entry":"Normalize","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/diffusionmodules/model.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/diffusionmodules/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3a6b977022957cb"}},{"code_sha256_prefix":"7439a9d4c6be04b0","entry":"adopt_weight","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/losses/vqperceptual.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/losses/vqperceptual.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7439a9d4c6be04b0"}},{"code_sha256_prefix":"85f67e8fb251251f","entry":"combine_contexts","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/discriminator/stylegan.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/discriminator/stylegan.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85f67e8fb251251f"}},{"code_sha256_prefix":"feab88ece44cfb43","entry":"default","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/discriminator/stylegan.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/discriminator/stylegan.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"feab88ece44cfb43"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"models/stage2/utils.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/models/stage2/utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/discriminator/stylegan.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/discriminator/stylegan.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"cb49209c125de1b4","entry":"get_timestep_embedding","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/diffusionmodules/model.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/diffusionmodules/model.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb49209c125de1b4"}},{"code_sha256_prefix":"22a83df891d750c6","entry":"hinge_d_loss","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/losses/vqperceptual.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/losses/vqperceptual.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22a83df891d750c6"}},{"code_sha256_prefix":"3137073275f8c21a","entry":"nonlinearity","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/diffusionmodules/model.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/diffusionmodules/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3137073275f8c21a"}},{"code_sha256_prefix":"7257fdf155f5ef45","entry":"normalize_tensor","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/losses/lpips.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/losses/lpips.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7257fdf155f5ef45"}},{"code_sha256_prefix":"e268053216b1dd62","entry":"spatial_average","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/losses/lpips.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/losses/lpips.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e268053216b1dd62"}},{"code_sha256_prefix":"263c54290c553ddd","entry":"vanilla_d_loss","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"modules/losses/vqperceptual.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/modules/losses/vqperceptual.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"263c54290c553ddd"}},{"code_sha256_prefix":"c9a9623e2ceb5de2","entry":"Scheduler_LinearWarmup","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"models/stage1/utils.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/models/stage1/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9a9623e2ceb5de2"}},{"code_sha256_prefix":"b0d5bd17b550d589","entry":"fn","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"models/stage2/utils.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/models/stage2/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0d5bd17b550d589"}},{"code_sha256_prefix":"524542d2884463d4","entry":"fn_LinearWarmup","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"models/stage1/utils.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/models/stage1/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"524542d2884463d4"}},{"code_sha256_prefix":"593ed2ee1ff174b3","entry":"fn_LinearWarmup_CosineDecay","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"models/stage1/utils.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/models/stage1/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"593ed2ee1ff174b3"}},{"code_sha256_prefix":"a7f95a154e8faecc","entry":"learning_rate_schedule","repo":"crossmodalgroup/maskedvectorquantization","repo_kind":"official","path":"models/stage2/utils.py","file_url":"https://github.com/crossmodalgroup/maskedvectorquantization/blob/HEAD/models/stage2/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7f95a154e8faecc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}