{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/controlar-controllable-image-generation-with","title":"ControlAR: Controllable Image Generation with Autoregressive Models","arxiv_id":"2410.02705","date":"2024-10-03","proceeding":null,"authors":["Zongming Li","Tianheng Cheng","Shoufa Chen","Peize Sun","Haocheng Shen","Longjin Ran","Xiaoxin Chen","Wenyu Liu","Xinggang Wang"],"abstract":"Autoregressive (AR) models have reformulated image generation as next-token prediction, demonstrating remarkable potential and emerging as strong competitors to diffusion models. However, control-to-image generation, akin to ControlNet, remains largely unexplored within AR models. Although a natural approach, inspired by advancements in Large Language Models, is to tokenize control images into tokens and prefill them into the autoregressive model before decoding image tokens, it still falls short in generation quality compared to ControlNet and suffers from inefficiency. To this end, we introduce ControlAR, an efficient and effective framework for integrating spatial controls into autoregressive image generation models. Firstly, we explore control encoding for AR models and propose a lightweight control encoder to transform spatial inputs (e.g., canny edges or depth maps) into control tokens. Then ControlAR exploits the conditional decoding method to generate the next image token conditioned on the per-token fusion between control and image tokens, similar to positional encodings. Compared to prefilling tokens, using conditional decoding significantly strengthens the control capability of AR models but also maintains the model's efficiency. Furthermore, the proposed ControlAR surprisingly empowers AR models with arbitrary-resolution image generation via conditional decoding and specific controls. Extensive experiments can demonstrate the controllability of the proposed ControlAR for the autoregressive control-to-image generation across diverse inputs, including edges, depths, and segmentation masks. Furthermore, both quantitative and qualitative results indicate that ControlAR surpasses previous state-of-the-art controllable diffusion models, e.g., ControlNet++. Code, models, and demo will soon be available at https://github.com/hustvl/ControlAR.","url_abs":"https://arxiv.org/abs/2410.02705v2","url_pdf":"https://arxiv.org/pdf/2410.02705v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"controlar-controllable-image-generation-with","repo_url":"https://github.com/hustvl/controlar","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.02705","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.02705"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/hustvl/ControlAR","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hustvl/controlar","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":12,"ran_honours":2,"ran_violates":1,"unverified":9},"by_repo_kind":{"official":{"samples":24,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0a1aa4e4837aba74","entry":"CaptionEmbedder","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0a1aa4e4837aba74"}},{"code_sha256_prefix":"28672d2f5d6d75b9","entry":"ConditionEmbedder","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"28672d2f5d6d75b9"}},{"code_sha256_prefix":"76f9e42efc0d4711","entry":"DropPath","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"76f9e42efc0d4711"}},{"code_sha256_prefix":"cf99c2811308e5e5","entry":"FeedForward","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf99c2811308e5e5"}},{"code_sha256_prefix":"0b441179596bf2b5","entry":"KVCache","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b441179596bf2b5"}},{"code_sha256_prefix":"7a710e6cde2ae35c","entry":"LabelEmbedder","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7a710e6cde2ae35c"}},{"code_sha256_prefix":"eec94bbc10306be0","entry":"MLP","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eec94bbc10306be0"}},{"code_sha256_prefix":"9e7e8b336fa1eb12","entry":"ModelArgs","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e7e8b336fa1eb12"}},{"code_sha256_prefix":"edd6b8543cc0d1ab","entry":"RMSNorm","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"edd6b8543cc0d1ab"}},{"code_sha256_prefix":"489c6f42beab887e","entry":"build_imagenet_code","repo":"hustvl/ControlAR","repo_kind":"official","path":"dataset/imagenet.py","file_url":"https://github.com/hustvl/ControlAR/blob/HEAD/dataset/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"489c6f42beab887e"}},{"code_sha256_prefix":"432162d12f959985","entry":"create_npz_from_sample_folder","repo":"hustvl/ControlAR","repo_kind":"official","path":"create_npz.py","file_url":"https://github.com/hustvl/ControlAR/blob/HEAD/create_npz.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"432162d12f959985"}},{"code_sha256_prefix":"f6ff7671338c9c92","entry":"find_multiple","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6ff7671338c9c92"}},{"code_sha256_prefix":"8e142bd1a80ed2ff","entry":"get_causal_mask","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8e142bd1a80ed2ff"}},{"code_sha256_prefix":"7f53b2cdcd16fad9","entry":"precompute_freqs_cis_2d","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f53b2cdcd16fad9"}},{"code_sha256_prefix":"2c1c9c90fbedc5d3","entry":"sample","repo":"hustvl/ControlAR","repo_kind":"official","path":"autoregressive/models/generate.py","file_url":"https://github.com/hustvl/ControlAR/blob/HEAD/autoregressive/models/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c1c9c90fbedc5d3"}},{"code_sha256_prefix":"8320019fce428945","entry":"Attention","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8320019fce428945"}},{"code_sha256_prefix":"152a894d1340f06b","entry":"Transformer","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"152a894d1340f06b"}},{"code_sha256_prefix":"7baa685224160fe5","entry":"TransformerBlock","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7baa685224160fe5"}},{"code_sha256_prefix":"7ea7bc571b41dc22","entry":"ViT_Adapter","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7ea7bc571b41dc22"}},{"code_sha256_prefix":"29173bf6147634f3","entry":"apply_rotary_emb","repo":"hustvl/controlar","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/controlar/blob/HEAD/autoregressive/models/gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29173bf6147634f3"}},{"code_sha256_prefix":"5cceaa9d0edbce06","entry":"build_imagenet","repo":"hustvl/ControlAR","repo_kind":"official","path":"dataset/imagenet.py","file_url":"https://github.com/hustvl/ControlAR/blob/HEAD/dataset/imagenet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5cceaa9d0edbce06"}},{"code_sha256_prefix":"9dac59261197935f","entry":"logits_to_probs","repo":"hustvl/ControlAR","repo_kind":"official","path":"autoregressive/models/generate.py","file_url":"https://github.com/hustvl/ControlAR/blob/HEAD/autoregressive/models/generate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9dac59261197935f"}},{"code_sha256_prefix":"0f0ff4e443413018","entry":"precompute_freqs_cis","repo":"hustvl/ControlAR","repo_kind":"official","path":"autoregressive/models/gpt.py","file_url":"https://github.com/hustvl/ControlAR/blob/HEAD/autoregressive/models/gpt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f0ff4e443413018"}},{"code_sha256_prefix":"f2fea48028c7ae80","entry":"top_k_top_p_filtering","repo":"hustvl/ControlAR","repo_kind":"official","path":"autoregressive/models/generate.py","file_url":"https://github.com/hustvl/ControlAR/blob/HEAD/autoregressive/models/generate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f2fea48028c7ae80"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}