{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/graph-structured-referring-expression","title":"Graph-Structured Referring Expression Reasoning in The Wild","arxiv_id":"2004.08814","date":"2020-04-19","proceeding":"CVPR 2020 6","authors":["Sibei Yang","Guanbin Li","Yizhou Yu"],"abstract":"Grounding referring expressions aims to locate in an image an object referred to by a natural language expression. The linguistic structure of a referring expression provides a layout of reasoning over the visual contents, and it is often crucial to align and jointly understand the image and the referring expression. In this paper, we propose a scene graph guided modular network (SGMN), which performs reasoning over a semantic graph and a scene graph with neural modules under the guidance of the linguistic structure of the expression. In particular, we model the image as a structured semantic graph, and parse the expression into a language scene graph. The language scene graph not only decodes the linguistic structure of the expression, but also has a consistent representation with the image semantic graph. In addition to exploring structured solutions to grounding referring expressions, we also propose Ref-Reasoning, a large-scale real-world dataset for structured referring expression reasoning. We automatically generate referring expressions over the scene graphs of images using diverse expression templates and functional programs. This dataset is equipped with real-world visual contents as well as semantically rich expressions with different reasoning layouts. Experimental results show that our SGMN not only significantly outperforms existing state-of-the-art algorithms on the new Ref-Reasoning dataset, but also surpasses state-of-the-art structured methods on commonly used benchmark datasets. It can also provide interpretable visual evidences of reasoning. Data and code are available at https://github.com/sibeiyang/sgmn","url_abs":"https://arxiv.org/abs/2004.08814v1","url_pdf":"https://arxiv.org/pdf/2004.08814v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"graph-structured-referring-expression","repo_url":"https://github.com/sibeiyang/sgmn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"referring-expression","task_name":"Referring Expression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.08814","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.08814"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sibeiyang/sgmn","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f6b888b1d5314ed","entry":"compute_IoU","repo":"sibeiyang/sgmn","repo_kind":"official","path":"lib/datasets/refdataset.py","file_url":"https://github.com/sibeiyang/sgmn/blob/HEAD/lib/datasets/refdataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f6b888b1d5314ed"}},{"code_sha256_prefix":"cb6005952bd7770e","entry":"get_db","repo":"sibeiyang/sgmn","repo_kind":"official","path":"lib/datasets/factory.py","file_url":"https://github.com/sibeiyang/sgmn/blob/HEAD/lib/datasets/factory.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb6005952bd7770e"}},{"code_sha256_prefix":"328805c34aaae128","entry":"get_split_projection","repo":"sibeiyang/sgmn","repo_kind":"official","path":"lib/datasets/refdataset_sg_online.py","file_url":"https://github.com/sibeiyang/sgmn/blob/HEAD/lib/datasets/refdataset_sg_online.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"328805c34aaae128"}},{"code_sha256_prefix":"8389e17b0d4fe924","entry":"max_overlap","repo":"sibeiyang/sgmn","repo_kind":"official","path":"lib/datasets/refdataset.py","file_url":"https://github.com/sibeiyang/sgmn/blob/HEAD/lib/datasets/refdataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8389e17b0d4fe924"}},{"code_sha256_prefix":"0ac073870d1f949d","entry":"random_hard_negative","repo":"sibeiyang/sgmn","repo_kind":"official","path":"lib/crits/criterion.py","file_url":"https://github.com/sibeiyang/sgmn/blob/HEAD/lib/crits/criterion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ac073870d1f949d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}