{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/relation-aware-instance-refinement-for-weakly","title":"Relation-aware Instance Refinement for Weakly Supervised Visual Grounding","arxiv_id":"2103.12989","date":"2021-03-24","proceeding":"CVPR 2021 1","authors":["Yongfei Liu","Bo Wan","Lin Ma","Xuming He"],"abstract":"Visual grounding, which aims to build a correspondence between visual objects and their language entities, plays a key role in cross-modal scene understanding. One promising and scalable strategy for learning visual grounding is to utilize weak supervision from only image-caption pairs. Previous methods typically rely on matching query phrases directly to a precomputed, fixed object candidate pool, which leads to inaccurate localization and ambiguous matching due to lack of semantic relation constraints. In our paper, we propose a novel context-aware weakly-supervised learning method that incorporates coarse-to-fine object refinement and entity relation modeling into a two-stage deep network, capable of producing more accurate object representation and matching. To effectively train our network, we introduce a self-taught regression loss for the proposal locations and a classification loss based on parsed entity relations. Extensive experiments on two public benchmarks Flickr30K Entities and ReferItGame demonstrate the efficacy of our weakly grounding framework. The results show that we outperform the previous methods by a considerable margin, achieving 59.27\\% top-1 accuracy in Flickr30K Entities and 37.68\\% in the ReferItGame dataset respectively (Code is available at https://github.com/youngfly11/ReIR-WeaklyGrounding.pytorch.git).","url_abs":"https://arxiv.org/abs/2103.12989v1","url_pdf":"https://arxiv.org/pdf/2103.12989v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"relation-aware-instance-refinement-for-weakly","repo_url":"https://github.com/youngfly11/ReIR-WeaklyGrounding.pytorch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"object","task_name":"Object"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"},{"task_slug":"weakly-supervised-learning","task_name":"Weakly-supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.12989","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.12989"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/youngfly11/ReIR-WeaklyGrounding.pytorch","reach":null}],"summary":{"ran":1,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"f552c5fdba0f4264","entry":"PhraseEmbeddingSent","repo":"youngfly11/ReIR-WeaklyGrounding.pytorch","repo_kind":"official","path":"detectron2/modeling/weaklygrounding/kac_net.py","file_url":"https://github.com/youngfly11/ReIR-WeaklyGrounding.pytorch/blob/HEAD/detectron2/modeling/weaklygrounding/kac_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f552c5fdba0f4264"}},{"code_sha256_prefix":"204847e91c2e6a2b","entry":"move2cpu","repo":"youngfly11/ReIR-WeaklyGrounding.pytorch","repo_kind":"official","path":"detectron2/modeling/weaklygrounding/kac_net.py","file_url":"https://github.com/youngfly11/ReIR-WeaklyGrounding.pytorch/blob/HEAD/detectron2/modeling/weaklygrounding/kac_net.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"204847e91c2e6a2b"}},{"code_sha256_prefix":"dddc4896a81c2ee8","entry":"numerical_stability_masked_softmax","repo":"youngfly11/ReIR-WeaklyGrounding.pytorch","repo_kind":"official","path":"detectron2/modeling/weaklygrounding/kac_net.py","file_url":"https://github.com/youngfly11/ReIR-WeaklyGrounding.pytorch/blob/HEAD/detectron2/modeling/weaklygrounding/kac_net.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dddc4896a81c2ee8"}},{"code_sha256_prefix":"050b382c184f7f05","entry":"KnowledgeAidedConsistencyNet","repo":"youngfly11/ReIR-WeaklyGrounding.pytorch","repo_kind":"official","path":"detectron2/modeling/weaklygrounding/kac_net.py","file_url":"https://github.com/youngfly11/ReIR-WeaklyGrounding.pytorch/blob/HEAD/detectron2/modeling/weaklygrounding/kac_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"050b382c184f7f05"}},{"code_sha256_prefix":"28e6fd5351f48136","entry":"WeaklyVGLossCompute","repo":"youngfly11/ReIR-WeaklyGrounding.pytorch","repo_kind":"official","path":"detectron2/modeling/weaklygrounding/kac_net.py","file_url":"https://github.com/youngfly11/ReIR-WeaklyGrounding.pytorch/blob/HEAD/detectron2/modeling/weaklygrounding/kac_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28e6fd5351f48136"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}