{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/illume-illuminating-unified-mllm-with-dual","title":"ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement","arxiv_id":"2504.01934","date":"2025-04-02","proceeding":null,"authors":["Runhui Huang","Chunwei Wang","Junwei Yang","Guansong Lu","Yunlong Yuan","Jianhua Han","Lu Hou","Wei zhang","Lanqing Hong","Hengshuang Zhao","Hang Xu"],"abstract":"We present ILLUME+ that leverages dual visual tokenization and a diffusion decoder to improve both deep semantic understanding and high-fidelity image generation. Existing unified models have struggled to simultaneously handle the three fundamental capabilities in a unified model: understanding, generation, and editing. Models like Chameleon and EMU3 utilize VQGAN for image discretization, due to the lack of deep semantic interaction, they lag behind specialist models like LLaVA in visual understanding tasks. To mitigate this, LaViT and ILLUME employ semantic encoders for tokenization, but they struggle with image editing due to poor texture preservation. Meanwhile, Janus series decouples the input and output image representation, limiting their abilities to seamlessly handle interleaved image-text understanding and generation. In contrast, ILLUME+ introduces a unified dual visual tokenizer, DualViTok, which preserves both fine-grained textures and text-aligned semantics while enabling a coarse-to-fine image representation strategy for multimodal understanding and generation. Additionally, we employ a diffusion model as the image detokenizer for enhanced generation quality and efficient super-resolution. ILLUME+ follows a continuous-input, discrete-output scheme within the unified MLLM and adopts a progressive training procedure that supports dynamic resolution across the vision tokenizer, MLLM, and diffusion decoder. This design allows for flexible and efficient context-aware image editing and generation across diverse tasks. ILLUME+ (3B) exhibits competitive performance against existing unified MLLMs and specialized models across multimodal understanding, generation, and editing benchmarks. With its strong performance, ILLUME+ provides a scalable and versatile foundation for future multimodal applications. Project Page: https://illume-unified-mllm.github.io/.","url_abs":"https://arxiv.org/abs/2504.01934v1","url_pdf":"https://arxiv.org/pdf/2504.01934v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"illume-illuminating-unified-mllm-with-dual","repo_url":"https://github.com/illume-unified-mllm/ILLUME_plus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-reconstruction","task_name":"Image Reconstruction"},{"task_slug":"super-resolution","task_name":"Super-Resolution"},{"task_slug":"text-based-image-editing","task_name":"Text-based Image Editing"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.01934","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.01934"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/illume-unified-mllm/ILLUME_plus","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_violates":1,"ran_fixture":1,"unverified":16},"by_repo_kind":{"listed":{"samples":20,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/mm_utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"2a3280757ce5a198","entry":"reduce_dict","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/dist_utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/dist_utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a3280757ce5a198"}},{"code_sha256_prefix":"468eedeba67f1b00","entry":"resize_and_pad_image","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/mm_utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"468eedeba67f1b00"}},{"code_sha256_prefix":"55c32993da87759b","entry":"unpad_image","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/model/illume_arch.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/model/illume_arch.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55c32993da87759b"}},{"code_sha256_prefix":"1c8d976c39a57c9e","entry":"all_gather","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/dist_utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/dist_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1c8d976c39a57c9e"}},{"code_sha256_prefix":"d953291e013c1472","entry":"calculate_image_token_num","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/generation_eval/models/inference_utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/generation_eval/models/inference_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d953291e013c1472"}},{"code_sha256_prefix":"c255b91c2246dc37","entry":"check_image_token_num","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/app_hf.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/app_hf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c255b91c2246dc37"}},{"code_sha256_prefix":"4e189d5dad79f682","entry":"get_local_proc_group","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/dist_utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/dist_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e189d5dad79f682"}},{"code_sha256_prefix":"c55a1f7e4549fb3e","entry":"is_zero3_model","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/model/utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/model/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c55a1f7e4549fb3e"}},{"code_sha256_prefix":"9778219ab71b8d25","entry":"load_and_merge_config","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/registry_utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/registry_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9778219ab71b8d25"}},{"code_sha256_prefix":"3e40955600cc6631","entry":"load_py_config","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/registry_utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/registry_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e40955600cc6631"}},{"code_sha256_prefix":"14dac797f0c4b233","entry":"load_state_dict_maybe_zero_3","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/model/utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/model/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"14dac797f0c4b233"}},{"code_sha256_prefix":"10f7276899092b15","entry":"merge_dicts","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/registry_utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/registry_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10f7276899092b15"}},{"code_sha256_prefix":"d4d72d2f8ec8ae97","entry":"pad_sequence","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/app_hf.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/app_hf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d4d72d2f8ec8ae97"}},{"code_sha256_prefix":"0a6a1f6fb96cd47b","entry":"pad_sequence","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/generation_eval/models/illume.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/generation_eval/models/illume.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0a6a1f6fb96cd47b"}},{"code_sha256_prefix":"c6ccde0b6ee512a0","entry":"parse_interleaved_text_image","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/generation_eval/models/inference_utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/generation_eval/models/inference_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c6ccde0b6ee512a0"}},{"code_sha256_prefix":"d40f1727de55b957","entry":"replace_last","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/conversation.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d40f1727de55b957"}},{"code_sha256_prefix":"25c22c3cf79f39b6","entry":"replace_ratio_tags_in_text","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/conversation.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25c22c3cf79f39b6"}},{"code_sha256_prefix":"4e41538bc082c01f","entry":"replace_sequentially_regex","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/conversation.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e41538bc082c01f"}},{"code_sha256_prefix":"29d67c6717548a53","entry":"select_best_resolution","repo":"illume-unified-mllm/ILLUME_plus","repo_kind":"listed","path":"ILLUME/illume/mm_utils.py","file_url":"https://github.com/illume-unified-mllm/ILLUME_plus/blob/HEAD/ILLUME/illume/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29d67c6717548a53"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}