{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-autoregressive-visual-generation","title":"Improving Autoregressive Visual Generation with Cluster-Oriented Token Prediction","arxiv_id":"2501.00880","date":"2025-01-01","proceeding":"CVPR 2025 1","authors":["Teng Hu","Jiangning Zhang","Ran Yi","Jieyu Weng","Yabiao Wang","Xianfang Zeng","Zhucun Xue","Lizhuang Ma"],"abstract":"Employing LLMs for visual generation has recently become a research focus. However, the existing methods primarily transfer the LLM architecture to visual generation but rarely investigate the fundamental differences between language and vision. This oversight may lead to suboptimal utilization of visual generation capabilities within the LLM framework. In this paper, we explore the characteristics of visual embedding space under the LLM framework and discover that the correlation between visual embeddings can help achieve more stable and robust generation results. We present IAR, an Improved AutoRegressive Visual Generation Method that enhances the training efficiency and generation quality of LLM-based visual generation models. Firstly, we propose a Codebook Rearrangement strategy that uses balanced k-means clustering algorithm to rearrange the visual codebook into clusters, ensuring high similarity among visual features within each cluster. Leveraging the rearranged codebook, we propose a Cluster-oriented Cross-entropy Loss that guides the model to correctly predict the cluster where the token is located. This approach ensures that even if the model predicts the wrong token index, there is a high probability the predicted token is located in the correct cluster, which significantly enhances the generation quality and robustness. Extensive experiments demonstrate that our method consistently enhances the model training efficiency and performance from 100M to 1.4B, reducing the training time by half while achieving the same FID. Additionally, our approach can be applied to various LLM-based visual generation models and adheres to the scaling law, providing a promising direction for future research in LLM-based visual generation.","url_abs":"https://arxiv.org/abs/2501.00880v1","url_pdf":"https://arxiv.org/pdf/2501.00880v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"prediction","task_name":"Prediction"}],"methods":[{"method_slug":"k-means-clustering","method_name":"k-Means Clustering"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.00880","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.00880"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sjtuplayer/IAR","reach":null}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"6f2102734f185e9f","entry":"Attention","repo":"sjtuplayer/IAR","repo_kind":"found_in_text","path":"autoregressive/models/gpt.py","file_url":"https://github.com/sjtuplayer/IAR/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f2102734f185e9f"}},{"code_sha256_prefix":"a95da8af9932c3df","entry":"CaptionEmbedder","repo":"sjtuplayer/IAR","repo_kind":"found_in_text","path":"autoregressive/models/gpt.py","file_url":"https://github.com/sjtuplayer/IAR/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a95da8af9932c3df"}},{"code_sha256_prefix":"0266f2bd1a7c84d4","entry":"FeedForward","repo":"sjtuplayer/IAR","repo_kind":"found_in_text","path":"autoregressive/models/gpt.py","file_url":"https://github.com/sjtuplayer/IAR/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0266f2bd1a7c84d4"}},{"code_sha256_prefix":"5cdccc4f721bbecd","entry":"LabelEmbedder","repo":"sjtuplayer/IAR","repo_kind":"found_in_text","path":"autoregressive/models/gpt.py","file_url":"https://github.com/sjtuplayer/IAR/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5cdccc4f721bbecd"}},{"code_sha256_prefix":"553e64473745e011","entry":"MLP","repo":"sjtuplayer/IAR","repo_kind":"found_in_text","path":"autoregressive/models/gpt.py","file_url":"https://github.com/sjtuplayer/IAR/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"553e64473745e011"}},{"code_sha256_prefix":"7e3bfcbaeb7daf19","entry":"ModelArgs","repo":"sjtuplayer/IAR","repo_kind":"found_in_text","path":"autoregressive/models/gpt.py","file_url":"https://github.com/sjtuplayer/IAR/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e3bfcbaeb7daf19"}},{"code_sha256_prefix":"c93a2120748f0cc8","entry":"Transformer","repo":"sjtuplayer/IAR","repo_kind":"found_in_text","path":"autoregressive/models/gpt.py","file_url":"https://github.com/sjtuplayer/IAR/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c93a2120748f0cc8"}},{"code_sha256_prefix":"f03b5731f02b2a1e","entry":"TransformerBlock","repo":"sjtuplayer/IAR","repo_kind":"found_in_text","path":"autoregressive/models/gpt.py","file_url":"https://github.com/sjtuplayer/IAR/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f03b5731f02b2a1e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}