{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/simvg-a-simple-framework-for-visual-grounding","title":"SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion","arxiv_id":"2409.17531","date":"2024-09-26","proceeding":null,"authors":["Ming Dai","Lingfeng Yang","Yihao Xu","ZhenHua Feng","Wankou Yang"],"abstract":"Visual grounding is a common vision task that involves grounding descriptive sentences to the corresponding regions of an image. Most existing methods use independent image-text encoding and apply complex hand-crafted modules or encoder-decoder architectures for modal interaction and query reasoning. However, their performance significantly drops when dealing with complex textual expressions. This is because the former paradigm only utilizes limited downstream data to fit the multi-modal feature fusion. Therefore, it is only effective when the textual expressions are relatively simple. In contrast, given the wide diversity of textual expressions and the uniqueness of downstream training data, the existing fusion module, which extracts multimodal content from a visual-linguistic context, has not been fully investigated. In this paper, we present a simple yet robust transformer-based framework, SimVG, for visual grounding. Specifically, we decouple visual-linguistic feature fusion from downstream tasks by leveraging existing multimodal pre-trained models and incorporating additional object tokens to facilitate deep integration of downstream and pre-training tasks. Furthermore, we design a dynamic weight-balance distillation method in the multi-branch synchronous learning process to enhance the representation capability of the simpler branch. This branch only consists of a lightweight MLP, which simplifies the structure and improves reasoning speed. Experiments on six widely used VG datasets, i.e., RefCOCO/+/g, ReferIt, Flickr30K, and GRefCOCO, demonstrate the superiority of SimVG. Finally, the proposed method not only achieves improvements in efficiency and convergence speed but also attains new state-of-the-art performance on these benchmarks. Codes and models will be available at \\url{https://github.com/Dmmm1997/SimVG}.","url_abs":"https://arxiv.org/abs/2409.17531v2","url_pdf":"https://arxiv.org/pdf/2409.17531v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"simvg-a-simple-framework-for-visual-grounding","repo_url":"https://github.com/dmmm1997/simvg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"generalized-referring-expression","task_name":"Generalized Referring Expression Comprehension"},{"task_slug":"referring-expression-comprehension","task_name":"Referring Expression Comprehension"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/generalized-referring-expression","task":"Generalized Referring Expression Comprehension","dataset":"gRefCOCO","model":"SimVG-DB","rank_in_archive_order":1,"of":5,"metrics":{"N-acc.":"54.7","Precision@(F1=1, IoU≥0.5)":"62.1"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.17531","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.17531"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dmmm1997/simvg","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":2,"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e0a06ded5d4f6c3c","entry":"box_cxcywh_to_xyxy","repo":"dmmm1997/simvg","repo_kind":"official","path":"simvg/core/utils.py","file_url":"https://github.com/dmmm1997/simvg/blob/HEAD/simvg/core/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0a06ded5d4f6c3c"}},{"code_sha256_prefix":"031b02fb990284ba","entry":"box_xyxy_to_cxcywh","repo":"dmmm1997/simvg","repo_kind":"official","path":"simvg/core/utils.py","file_url":"https://github.com/dmmm1997/simvg/blob/HEAD/simvg/core/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"031b02fb990284ba"}},{"code_sha256_prefix":"2237f6e68317d7ae","entry":"gen_cam","repo":"dmmm1997/simvg","repo_kind":"official","path":"simvg/utils/gradcam_models.py","file_url":"https://github.com/dmmm1997/simvg/blob/HEAD/simvg/utils/gradcam_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2237f6e68317d7ae"}},{"code_sha256_prefix":"8173e275f8ad28a5","entry":"norm_image","repo":"dmmm1997/simvg","repo_kind":"official","path":"simvg/utils/gradcam_models.py","file_url":"https://github.com/dmmm1997/simvg/blob/HEAD/simvg/utils/gradcam_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8173e275f8ad28a5"}},{"code_sha256_prefix":"67fafc5d7d7e7875","entry":"parse_conv_block","repo":"dmmm1997/simvg","repo_kind":"official","path":"simvg/models/utils.py","file_url":"https://github.com/dmmm1997/simvg/blob/HEAD/simvg/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67fafc5d7d7e7875"}},{"code_sha256_prefix":"a5163995778e149c","entry":"parse_yolo_block","repo":"dmmm1997/simvg","repo_kind":"official","path":"simvg/models/utils.py","file_url":"https://github.com/dmmm1997/simvg/blob/HEAD/simvg/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5163995778e149c"}},{"code_sha256_prefix":"dff0f0000acdc0ad","entry":"sigmoid_focal_loss","repo":"dmmm1997/simvg","repo_kind":"official","path":"simvg/core/criterion/criterion.py","file_url":"https://github.com/dmmm1997/simvg/blob/HEAD/simvg/core/criterion/criterion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dff0f0000acdc0ad"}},{"code_sha256_prefix":"3189378f2936393a","entry":"build_matcher","repo":"dmmm1997/simvg","repo_kind":"official","path":"simvg/core/matcher.py","file_url":"https://github.com/dmmm1997/simvg/blob/HEAD/simvg/core/matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3189378f2936393a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}