{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multi-task-visual-grounding-with-coarse-to","title":"Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints","arxiv_id":"2501.06710","date":"2025-01-12","proceeding":null,"authors":["Ming Dai","Jian Li","Jiedong Zhuang","Xian Zhang","Wankou Yang"],"abstract":"Multi-task visual grounding involves the simultaneous execution of localization and segmentation in images based on textual expressions. The majority of advanced methods predominantly focus on transformer-based multimodal fusion, aiming to extract robust multimodal representations. However, ambiguity between referring expression comprehension (REC) and referring image segmentation (RIS) is error-prone, leading to inconsistencies between multi-task predictions. Besides, insufficient multimodal understanding directly contributes to biased target perception. To overcome these challenges, we propose a Coarse-to-fine Consistency Constraints Visual Grounding architecture ($\\text{C}^3\\text{VG}$), which integrates implicit and explicit modeling approaches within a two-stage framework. Initially, query and pixel decoders are employed to generate preliminary detection and segmentation outputs, a process referred to as the Rough Semantic Perception (RSP) stage. These coarse predictions are subsequently refined through the proposed Mask-guided Interaction Module (MIM) and a novel explicit bidirectional consistency constraint loss to ensure consistent representations across tasks, which we term the Refined Consistency Interaction (RCI) stage. Furthermore, to address the challenge of insufficient multimodal understanding, we leverage pre-trained models based on visual-linguistic fusion representations. Empirical evaluations on the RefCOCO, RefCOCO+, and RefCOCOg datasets demonstrate the efficacy and soundness of $\\text{C}^3\\text{VG}$, which significantly outperforms state-of-the-art REC and RIS methods by a substantial margin. Code and model will be available at \\url{https://github.com/Dmmm1997/C3VG}.","url_abs":"https://arxiv.org/abs/2501.06710v1","url_pdf":"https://arxiv.org/pdf/2501.06710v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-task-visual-grounding-with-coarse-to","repo_url":"https://github.com/dmmm1997/c3vg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"referring-expression","task_name":"Referring Expression"},{"task_slug":"referring-expression-comprehension","task_name":"Referring Expression Comprehension"},{"task_slug":"referring-expression-segmentation","task_name":"Referring Expression Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-8","task":"Referring Expression Segmentation","dataset":"RefCOCO testA","model":"C3VG","rank_in_archive_order":5,"of":13,"metrics":{"Overall IoU":"83.18"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-9","task":"Referring Expression Segmentation","dataset":"RefCOCO testB","model":"C3VG","rank_in_archive_order":6,"of":13,"metrics":{"Overall IoU":"77.86"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-5","task":"Referring Expression Segmentation","dataset":"RefCOCO+ test B","model":"C3VG","rank_in_archive_order":5,"of":30,"metrics":{"Overall IoU":"68.95"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-4","task":"Referring Expression Segmentation","dataset":"RefCOCO+ testA","model":"C3VG","rank_in_archive_order":7,"of":30,"metrics":{"Overall IoU":"77.96"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-3","task":"Referring Expression Segmentation","dataset":"RefCOCO+ val","model":"C3VG","rank_in_archive_order":6,"of":33,"metrics":{"Overall IoU":"74.68"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-segmentation-on-refcocog-1","task":"Referring Expression Segmentation","dataset":"RefCOCOg-test","model":"C3VG","rank_in_archive_order":6,"of":18,"metrics":{"Overall IoU":"76.39"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-segmentation-on-refcocog","task":"Referring Expression Segmentation","dataset":"RefCOCOg-val","model":"C3VG","rank_in_archive_order":8,"of":23,"metrics":{"Overall IoU":"74.43"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco","task":"Referring Expression Segmentation","dataset":"RefCoCo val","model":"C3VG","rank_in_archive_order":10,"of":37,"metrics":{"Overall IoU":"80.89"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.06710","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.06710"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dmmm1997/c3vg","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":3,"ran_honours":1,"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"official":{"samples":13,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c8ea5508b3ca2db0","entry":"apply_box_noise","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/layers/denoising.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/layers/denoising.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c8ea5508b3ca2db0"}},{"code_sha256_prefix":"95bd488170c50d24","entry":"apply_label_noise","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/layers/denoising.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/layers/denoising.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"95bd488170c50d24"}},{"code_sha256_prefix":"e0a06ded5d4f6c3c","entry":"box_cxcywh_to_xyxy","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/core/utils.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/core/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0a06ded5d4f6c3c"}},{"code_sha256_prefix":"856982244cffd7e3","entry":"box_cxcywh_to_xyxy","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/layers/box_ops.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/layers/box_ops.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"856982244cffd7e3"}},{"code_sha256_prefix":"be924e54ed79731b","entry":"multi_scale_deformable_attn_pytorch","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/layers/multi_scale_deform_attn.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/layers/multi_scale_deform_attn.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be924e54ed79731b"}},{"code_sha256_prefix":"abe66849ff8e9119","entry":"box_iou","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/layers/box_ops.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/layers/box_ops.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"abe66849ff8e9119"}},{"code_sha256_prefix":"9d703502a5c06dd4","entry":"box_xyxy_to_cxcywh","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/layers/box_ops.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/layers/box_ops.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d703502a5c06dd4"}},{"code_sha256_prefix":"c8f336bbb9a19971","entry":"build_act_layer","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/layers/dcn_v3.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/layers/dcn_v3.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c8f336bbb9a19971"}},{"code_sha256_prefix":"3189378f2936393a","entry":"build_matcher","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/core/matcher.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/core/matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3189378f2936393a"}},{"code_sha256_prefix":"132d865c3c70a243","entry":"build_norm_layer","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/layers/dcn_v3.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/layers/dcn_v3.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"132d865c3c70a243"}},{"code_sha256_prefix":"47b44963c80f19d1","entry":"create_dummy_class","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/layers/multi_scale_deform_attn.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/layers/multi_scale_deform_attn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47b44963c80f19d1"}},{"code_sha256_prefix":"5f2ed565ef0da8fc","entry":"create_dummy_func","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/layers/multi_scale_deform_attn.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/layers/multi_scale_deform_attn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f2ed565ef0da8fc"}},{"code_sha256_prefix":"a63400f18f4a819c","entry":"dcnv3_core_pytorch","repo":"dmmm1997/c3vg","repo_kind":"official","path":"c3vg/layers/dcn_v3.py","file_url":"https://github.com/dmmm1997/c3vg/blob/HEAD/c3vg/layers/dcn_v3.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a63400f18f4a819c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}