{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/referring-relationships","title":"Referring Relationships","arxiv_id":"1803.10362","date":"2018-03-28","proceeding":"CVPR 2018 6","authors":["Ranjay Krishna","Ines Chami","Michael Bernstein","Li Fei-Fei"],"abstract":"Images are not simply sets of objects: each image represents a web of\ninterconnected relationships. These relationships between entities carry\nsemantic meaning and help a viewer differentiate between instances of an\nentity. For example, in an image of a soccer match, there may be multiple\npersons present, but each participates in different relationships: one is\nkicking the ball, and the other is guarding the goal. In this paper, we\nformulate the task of utilizing these \"referring relationships\" to disambiguate\nbetween entities of the same category. We introduce an iterative model that\nlocalizes the two entities in the referring relationship, conditioned on one\nanother. We formulate the cyclic condition between the entities in a\nrelationship by modelling predicates that connect the entities as shifts in\nattention from one entity to another. We demonstrate that our model can not\nonly outperform existing approaches on three datasets --- CLEVR, VRD and Visual\nGenome --- but also that it produces visually meaningful predicate shifts, as\nan instance of interpretable neural networks. Finally, we show that by\nmodelling predicates as attention shifts, we can even localize entities in the\nabsence of their category, allowing our model to find completely unseen\ncategories.","url_abs":"http://arxiv.org/abs/1803.10362v2","url_pdf":"http://arxiv.org/pdf/1803.10362v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"referring-relationships","repo_url":"https://github.com/StanfordVL/ReferringRelationships","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"referring-relationships","repo_url":"https://github.com/shikorab/DSG","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1803.10362","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}