{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/controlmllm-training-free-visual-prompt","title":"ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models","arxiv_id":"2407.21534","date":"2024-07-31","proceeding":null,"authors":["Mingrui Wu","Xinyue Cai","Jiayi Ji","Jiale Li","Oucheng Huang","Gen Luo","Hao Fei","Guannan Jiang","Xiaoshuai Sun","Rongrong Ji"],"abstract":"In this work, we propose a training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs) through test-time optimization of a learnable latent variable. We observe that attention, as the core module of MLLMs, connects text prompt tokens and visual tokens, ultimately determining the final results. Our approach involves adjusting visual tokens from the MLP output at test time, controlling the attention response to ensure text prompt tokens attend to visual tokens in referring regions. We optimize a learnable latent variable based on an energy function, enhancing the strength of referring regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referring abilities into MLLMs, and supports referring with box, mask, scribble and point. The results demonstrate that our method exhibits out-of-domain generalization and interpretability.","url_abs":"https://arxiv.org/abs/2407.21534v6","url_pdf":"https://arxiv.org/pdf/2407.21534v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"controlmllm-training-free-visual-prompt","repo_url":"https://github.com/mrwu-mac/controlmllm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"prompt-learning","task_name":"Prompt Learning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.21534","atlas_url":"https://app.syntology.ai/?focus=2407.21534","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.21534"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mrwu-mac/controlmllm","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"75d76a38bbfdc998","entry":"att_window_from_bbox","repo":"mrwu-mac/controlmllm","repo_kind":"official","path":"controlmllm++/qwen2_5_vl/qwen_utils.py","file_url":"https://github.com/mrwu-mac/controlmllm/blob/HEAD/controlmllm%2B%2B/qwen2_5_vl/qwen_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75d76a38bbfdc998"}},{"code_sha256_prefix":"ac99733f80f7b69e","entry":"build_mask_from_bbox","repo":"mrwu-mac/controlmllm","repo_kind":"official","path":"controlmllm++/qwen2_5_vl/qwen_utils.py","file_url":"https://github.com/mrwu-mac/controlmllm/blob/HEAD/controlmllm%2B%2B/qwen2_5_vl/qwen_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac99733f80f7b69e"}},{"code_sha256_prefix":"62f41cf13196989a","entry":"compute_ca_loss","repo":"mrwu-mac/controlmllm","repo_kind":"official","path":"controlmllm++/llava/utils.py","file_url":"https://github.com/mrwu-mac/controlmllm/blob/HEAD/controlmllm%2B%2B/llava/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62f41cf13196989a"}},{"code_sha256_prefix":"18de0523aa840ac7","entry":"compute_ca_loss","repo":"mrwu-mac/controlmllm","repo_kind":"official","path":"controlmllm/instructblip/utils.py","file_url":"https://github.com/mrwu-mac/controlmllm/blob/HEAD/controlmllm/instructblip/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"18de0523aa840ac7"}},{"code_sha256_prefix":"a9210833fb8a046d","entry":"gaussian","repo":"mrwu-mac/controlmllm","repo_kind":"official","path":"controlmllm++/llava/utils.py","file_url":"https://github.com/mrwu-mac/controlmllm/blob/HEAD/controlmllm%2B%2B/llava/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a9210833fb8a046d"}},{"code_sha256_prefix":"dd419e18dc676bcc","entry":"get_grid_shape","repo":"mrwu-mac/controlmllm","repo_kind":"official","path":"controlmllm++/qwen2_5_vl/qwen_utils.py","file_url":"https://github.com/mrwu-mac/controlmllm/blob/HEAD/controlmllm%2B%2B/qwen2_5_vl/qwen_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd419e18dc676bcc"}},{"code_sha256_prefix":"4c90a0e507c16145","entry":"show_image_relevance","repo":"mrwu-mac/controlmllm","repo_kind":"official","path":"controlmllm++/llava/utils.py","file_url":"https://github.com/mrwu-mac/controlmllm/blob/HEAD/controlmllm%2B%2B/llava/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c90a0e507c16145"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}