{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-latent-scene-graph-representations","title":"Differentiable Scene Graphs","arxiv_id":"1902.10200","date":"2019-02-26","proceeding":null,"authors":["Moshiko Raboh","Roei Herzig","Gal Chechik","Jonathan Berant","Amir Globerson"],"abstract":"Reasoning about complex visual scenes involves perception of entities and their relations. Scene graphs provide a natural representation for reasoning tasks, by assigning labels to both entities (nodes) and relations (edges). Unfortunately, reasoning systems based on SGs are typically trained in a two-step procedure: First, training a model to predict SGs from images; Then, a separate model is created to reason based on predicted SGs. In many domains, it is preferable to train systems jointly in an end-to-end manner, but SGs are not commonly used as intermediate components in visual reasoning systems because being discrete and sparse, scene-graph representations are non-differentiable and difficult to optimize. Here we propose Differentiable Scene Graphs (DSGs), an image representation that is amenable to differentiable end-to-end optimization, and requires supervision only from the downstream tasks. DSGs provide a dense representation for all regions and pairs of regions, and do not spend modelling capacity on areas of the images that do not contain objects or relations of interest. We evaluate our model on the challenging task of identifying referring relationships (RR) in three benchmark datasets, Visual Genome, VRD and CLEVR. We describe a multi-task objective, and train in an end-to-end manner supervised by the downstream RR task. Using DSGs as an intermediate representation leads to new state-of-the-art performance.","url_abs":"https://arxiv.org/abs/1902.10200v5","url_pdf":"https://arxiv.org/pdf/1902.10200v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-latent-scene-graph-representations","repo_url":"https://github.com/shikorab/DSG","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1902.10200","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1902.10200"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shikorab/DSG","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d4f2ade15edff997","entry":"generate_anchors","repo":"shikorab/DSG","repo_kind":"official","path":"lib/layer_utils/generate_anchors.py","file_url":"https://github.com/shikorab/DSG/blob/HEAD/lib/layer_utils/generate_anchors.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d4f2ade15edff997"}},{"code_sha256_prefix":"3be9228820b46ef3","entry":"get_imdb","repo":"shikorab/DSG","repo_kind":"official","path":"lib/datasets/factory.py","file_url":"https://github.com/shikorab/DSG/blob/HEAD/lib/datasets/factory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3be9228820b46ef3"}},{"code_sha256_prefix":"8a015c012f507631","entry":"unique_boxes","repo":"shikorab/DSG","repo_kind":"official","path":"lib/datasets/ds_utils.py","file_url":"https://github.com/shikorab/DSG/blob/HEAD/lib/datasets/ds_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a015c012f507631"}},{"code_sha256_prefix":"d3e019b11b709edb","entry":"xywh_to_xyxy","repo":"shikorab/DSG","repo_kind":"official","path":"lib/datasets/ds_utils.py","file_url":"https://github.com/shikorab/DSG/blob/HEAD/lib/datasets/ds_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3e019b11b709edb"}},{"code_sha256_prefix":"a832e5017f504280","entry":"xyxy_to_xywh","repo":"shikorab/DSG","repo_kind":"official","path":"lib/datasets/ds_utils.py","file_url":"https://github.com/shikorab/DSG/blob/HEAD/lib/datasets/ds_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a832e5017f504280"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}