{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spectrum-guided-multi-granularity-referring","title":"Spectrum-guided Multi-granularity Referring Video Object Segmentation","arxiv_id":"2307.13537","date":"2023-07-25","proceeding":"ICCV 2023 1","authors":["Bo Miao","Mohammed Bennamoun","Yongsheng Gao","Ajmal Mian"],"abstract":"Current referring video object segmentation (R-VOS) techniques extract conditional kernels from encoded (low-resolution) vision-language features to segment the decoded high-resolution features. We discovered that this causes significant feature drift, which the segmentation kernels struggle to perceive during the forward computation. This negatively affects the ability of segmentation kernels. To address the drift problem, we propose a Spectrum-guided Multi-granularity (SgMg) approach, which performs direct segmentation on the encoded features and employs visual details to further optimize the masks. In addition, we propose Spectrum-guided Cross-modal Fusion (SCF) to perform intra-frame global interactions in the spectral domain for effective multimodal representation. Finally, we extend SgMg to perform multi-object R-VOS, a new paradigm that enables simultaneous segmentation of multiple referred objects in a video. This not only makes R-VOS faster, but also more practical. Extensive experiments show that SgMg achieves state-of-the-art performance on four video benchmark datasets, outperforming the nearest competitor by 2.8% points on Ref-YouTube-VOS. Our extended SgMg enables multi-object R-VOS, runs about 3 times faster while maintaining satisfactory performance. Code is available at https://github.com/bo-miao/SgMg.","url_abs":"https://arxiv.org/abs/2307.13537v1","url_pdf":"https://arxiv.org/pdf/2307.13537v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spectrum-guided-multi-granularity-referring","repo_url":"https://github.com/bo-miao/sgmg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"object","task_name":"Object"},{"task_slug":"referring-expression-segmentation","task_name":"Referring Expression Segmentation"},{"task_slug":"referring-video-object-segmentation","task_name":"Referring Video Object Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"video-object-segmentation","task_name":"Video Object Segmentation"},{"task_slug":"video-semantic-segmentation","task_name":"Video Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/referring-expression-segmentation-on-a2d","task":"Referring Expression Segmentation","dataset":"A2D Sentences","model":"SgMg (Video-Swin-B)","rank_in_archive_order":1,"of":27,"metrics":{"AP":"0.585","IoU mean":"0.720","IoU overall":"0.799","Precision@0.5":"0.843","Precision@0.6":"0.822","Precision@0.7":"0.767","Precision@0.8":"0.617","Precision@0.9":"0.259"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-davis","task":"Referring Expression Segmentation","dataset":"DAVIS 2017 (val)","model":"SgMg","rank_in_archive_order":5,"of":18,"metrics":{"J&F 1st frame":"63.3"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-j-hmdb","task":"Referring Expression Segmentation","dataset":"J-HMDB","model":"SgMg (Video-Swin-B)","rank_in_archive_order":1,"of":21,"metrics":{"AP":"0.450","IoU mean":"0.725","IoU overall":"0.737","Precision@0.5":"0.972","Precision@0.6":"0.917","Precision@0.7":"0.714","Precision@0.8":"0.225","Precision@0.9":"0.003"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refer-1","task":"Referring Expression Segmentation","dataset":"Refer-YouTube-VOS (2021 public validation)","model":"SgMg (Pre-training)","rank_in_archive_order":15,"of":33,"metrics":{"F":"67.4","J":"63.9","J&F":"65.7"},"uses_additional_data":false},{"leaderboard":"/sota/referring-video-object-segmentation-on-ref","task":"Referring Video Object Segmentation","dataset":"Ref-DAVIS17","model":"SgMg","rank_in_archive_order":7,"of":11,"metrics":{"F":"66.0","J":"60.6","J&F":"63.3"},"uses_additional_data":false},{"leaderboard":"/sota/referring-video-object-segmentation-on-refer","task":"Referring Video Object Segmentation","dataset":"Refer-YouTube-VOS","model":"SgMg","rank_in_archive_order":7,"of":18,"metrics":{"F":"67.4","J":"63.9","J&F":"65.7"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.13537","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.13537"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bo-miao/sgmg","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/bo-miao/SgMg","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":1,"ran":4,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"29dbbaafa4be18e1","entry":"aligned_bilinear","repo":"bo-miao/sgmg","repo_kind":"official","path":"models/sgmg.py","file_url":"https://github.com/bo-miao/sgmg/blob/HEAD/models/sgmg.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"29dbbaafa4be18e1"}},{"code_sha256_prefix":"ef1a3e10a9dbf4a9","entry":"box_cxcywh_to_xyxy","repo":"bo-miao/sgmg","repo_kind":"official","path":"inference_davis.py","file_url":"https://github.com/bo-miao/sgmg/blob/HEAD/inference_davis.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef1a3e10a9dbf4a9"}},{"code_sha256_prefix":"ed1a308adb65f645","entry":"compute_locations","repo":"bo-miao/sgmg","repo_kind":"official","path":"models/sgmg.py","file_url":"https://github.com/bo-miao/sgmg/blob/HEAD/models/sgmg.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ed1a308adb65f645"}},{"code_sha256_prefix":"6296ea80154ae7f9","entry":"dice_coef","repo":"bo-miao/sgmg","repo_kind":"official","path":"models/matcher.py","file_url":"https://github.com/bo-miao/sgmg/blob/HEAD/models/matcher.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6296ea80154ae7f9"}},{"code_sha256_prefix":"2626a807bce7587a","entry":"rescale_bboxes","repo":"bo-miao/sgmg","repo_kind":"official","path":"inference_davis.py","file_url":"https://github.com/bo-miao/sgmg/blob/HEAD/inference_davis.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2626a807bce7587a"}},{"code_sha256_prefix":"9502374fe12f11e4","entry":"sigmoid_focal_coef","repo":"bo-miao/sgmg","repo_kind":"official","path":"models/matcher.py","file_url":"https://github.com/bo-miao/sgmg/blob/HEAD/models/matcher.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9502374fe12f11e4"}},{"code_sha256_prefix":"ce24c98e0f98c5ed","entry":"SgMg","repo":"bo-miao/SgMg","repo_kind":"official","path":"models/sgmg.py","file_url":"https://github.com/bo-miao/SgMg/blob/HEAD/models/sgmg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ce24c98e0f98c5ed"}},{"code_sha256_prefix":"af3dd7f95037a80f","entry":"parse_dynamic_params","repo":"bo-miao/sgmg","repo_kind":"official","path":"models/sgmg.py","file_url":"https://github.com/bo-miao/sgmg/blob/HEAD/models/sgmg.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af3dd7f95037a80f"}},{"code_sha256_prefix":"ccbe959ea69981ad","entry":"vis_add_mask","repo":"bo-miao/sgmg","repo_kind":"official","path":"inference_davis.py","file_url":"https://github.com/bo-miao/sgmg/blob/HEAD/inference_davis.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ccbe959ea69981ad"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}