{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-grounding-with-multi-modal-conditional","title":"Visual Grounding with Multi-modal Conditional Adaptation","arxiv_id":"2409.04999","date":"2024-09-08","proceeding":null,"authors":["Ruilin Yao","Shengwu Xiong","Yichen Zhao","Yi Rong"],"abstract":"Visual grounding is the task of locating objects specified by natural language expressions. Existing methods extend generic object detection frameworks to tackle this task. They typically extract visual and textual features separately using independent visual and textual encoders, then fuse these features in a multi-modal decoder for final prediction. However, visual grounding presents unique challenges. It often involves locating objects with different text descriptions within the same image. Existing methods struggle with this task because the independent visual encoder produces identical visual features for the same image, limiting detection performance. Some recently approaches propose various language-guided visual encoders to address this issue, but they mostly rely solely on textual information and require sophisticated designs. In this paper, we introduce Multi-modal Conditional Adaptation (MMCA), which enables the visual encoder to adaptively update weights, directing its focus towards text-relevant regions. Specifically, we first integrate information from different modalities to obtain multi-modal embeddings. Then we utilize a set of weighting coefficients, which generated from the multimodal embeddings, to reorganize the weight update matrices and apply them to the visual encoder of the visual grounding model. Extensive experiments on four widely used datasets demonstrate that MMCA achieves significant improvements and state-of-the-art results. Ablation experiments further demonstrate the lightweight and efficiency of our method. Our source code is available at: https://github.com/Mr-Bigworth/MMCA.","url_abs":"https://arxiv.org/abs/2409.04999v1","url_pdf":"https://arxiv.org/pdf/2409.04999v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visual-grounding-with-multi-modal-conditional","repo_url":"https://github.com/mr-bigworth/mmca","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.04999","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.04999"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mr-bigworth/mmca","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e0451d070cb0b43e","entry":"build_vl_transformer","repo":"mr-bigworth/mmca","repo_kind":"official","path":"models_mmca_vector_based/vl_transformer.py","file_url":"https://github.com/mr-bigworth/mmca/blob/HEAD/models_mmca_vector_based/vl_transformer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0451d070cb0b43e"}},{"code_sha256_prefix":"dd06382c371fad2d","entry":"build_transformer","repo":"mr-bigworth/mmca","repo_kind":"official","path":"models_mmca_vector_based/visual_model/transformer.py","file_url":"https://github.com/mr-bigworth/mmca/blob/HEAD/models_mmca_vector_based/visual_model/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd06382c371fad2d"}},{"code_sha256_prefix":"105ac2dc1417db62","entry":"yolo_loss","repo":"mr-bigworth/mmca","repo_kind":"official","path":"utils/loss_utils.py","file_url":"https://github.com/mr-bigworth/mmca/blob/HEAD/utils/loss_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"105ac2dc1417db62"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}