{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/machine-mental-imagery-empower-multimodal","title":"Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens","arxiv_id":"2506.17218","date":"2025-06-20","proceeding":null,"authors":["Zeyuan Yang","Xueyang Yu","Delin Chen","Maohao Shen","Chuang Gan"],"abstract":"Vision-language models (VLMs) excel at multimodal understanding, yet their text-only decoding forces them to verbalize visual reasoning, limiting performance on tasks that demand visual imagination. Recent attempts train VLMs to render explicit images, but the heavy image-generation pre-training often hinders the reasoning ability. Inspired by the way humans reason with mental imagery-the internal construction and manipulation of visual cues-we investigate whether VLMs can reason through interleaved multimodal trajectories without producing explicit images. To this end, we present a Machine Mental Imagery framework, dubbed as Mirage, which augments VLM decoding with latent visual tokens alongside ordinary text. Concretely, whenever the model chooses to ``think visually'', it recasts its hidden states as next tokens, thereby continuing a multimodal trajectory without generating pixel-level images. Begin by supervising the latent tokens through distillation from ground-truth image embeddings, we then switch to text-only supervision to make the latent trajectory align tightly with the task objective. A subsequent reinforcement learning stage further enhances the multimodal reasoning capability. Experiments on diverse benchmarks demonstrate that Mirage unlocks stronger multimodal reasoning without explicit image generation.","url_abs":"https://arxiv.org/abs/2506.17218v1","url_pdf":"https://arxiv.org/pdf/2506.17218v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"machine-mental-imagery-empower-multimodal","repo_url":"https://github.com/umass-embodied-agi/mirage","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"multimodal-reasoning","task_name":"Multimodal Reasoning"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.17218","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.17218"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/umass-embodied-agi/mirage","reach":{"status":"ok"}}],"summary":{"unverified":17},"by_repo_kind":{"official":{"samples":17,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"ab306165fa74656c","entry":"build_tip_message","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"transformers/utils/deprecate_models.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/transformers/utils/deprecate_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab306165fa74656c"}},{"code_sha256_prefix":"54b7273383fcfc2d","entry":"extract_model_info","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"transformers/utils/deprecate_models.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/transformers/utils/deprecate_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54b7273383fcfc2d"}},{"code_sha256_prefix":"b176efb26f72e507","entry":"get_architectures_from_config_class","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b176efb26f72e507"}},{"code_sha256_prefix":"8d6745e2692cf03d","entry":"get_cased_name","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"transformers/utils/modular_model_converter.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/transformers/utils/modular_model_converter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d6745e2692cf03d"}},{"code_sha256_prefix":"3d999b95009c7537","entry":"get_config_class_from_processor_class","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d999b95009c7537"}},{"code_sha256_prefix":"938fc7c2422ef91d","entry":"get_list_of_repo_model_paths","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"transformers/utils/models_to_deprecate.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/transformers/utils/models_to_deprecate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"938fc7c2422ef91d"}},{"code_sha256_prefix":"051d790d9b197d9d","entry":"get_model_doc_path","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"transformers/utils/deprecate_models.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/transformers/utils/deprecate_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"051d790d9b197d9d"}},{"code_sha256_prefix":"029be6c1f552852b","entry":"get_module_source_from_name","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"transformers/utils/modular_model_converter.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/transformers/utils/modular_model_converter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"029be6c1f552852b"}},{"code_sha256_prefix":"0f78c7037b35a137","entry":"get_new_python_files_between_commits","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"transformers/utils/pr_slow_ci_models.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/transformers/utils/pr_slow_ci_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f78c7037b35a137"}},{"code_sha256_prefix":"0c0b81b081919d8e","entry":"get_processor_types_from_config_class","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c0b81b081919d8e"}},{"code_sha256_prefix":"0fbb3cf4415efb41","entry":"load_jsonl_dataset","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fbb3cf4415efb41"}},{"code_sha256_prefix":"01f09967a9ef73ab","entry":"multiple_input_images_preprocess_function","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"src/task.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/src/task.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01f09967a9ef73ab"}},{"code_sha256_prefix":"67012056b57c9945","entry":"place_input_image","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67012056b57c9945"}},{"code_sha256_prefix":"a6823237c70a51ca","entry":"place_output_image","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a6823237c70a51ca"}},{"code_sha256_prefix":"d646d0a64800a8fe","entry":"preserve_case_replace","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"transformers/utils/modular_model_converter.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/transformers/utils/modular_model_converter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d646d0a64800a8fe"}},{"code_sha256_prefix":"b76973ab8137eb01","entry":"single_input_image_preprocess_function","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"src/task.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/src/task.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b76973ab8137eb01"}},{"code_sha256_prefix":"c9aa7df020dc5885","entry":"single_input_image_test_preprocess_function","repo":"umass-embodied-agi/mirage","repo_kind":"official","path":"src/task.py","file_url":"https://github.com/umass-embodied-agi/mirage/blob/HEAD/src/task.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c9aa7df020dc5885"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}