{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mapping-images-to-scene-graphs-with","title":"Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction","arxiv_id":"1802.05451","date":"2018-02-15","proceeding":"NeurIPS 2018 12","authors":["Roei Herzig","Moshiko Raboh","Gal Chechik","Jonathan Berant","Amir Globerson"],"abstract":"Machine understanding of complex images is a key goal of artificial\nintelligence. One challenge underlying this task is that visual scenes contain\nmultiple inter-related objects, and that global context plays an important role\nin interpreting the scene. A natural modeling framework for capturing such\neffects is structured prediction, which optimizes over complex labels, while\nmodeling within-label interactions. However, it is unclear what principles\nshould guide the design of a structured prediction model that utilizes the\npower of deep learning components. Here we propose a design principle for such\narchitectures that follows from a natural requirement of permutation\ninvariance. We prove a necessary and sufficient characterization for\narchitectures that follow this invariance, and discuss its implication on model\ndesign. Finally, we show that the resulting model achieves new state of the art\nresults on the Visual Genome scene graph labeling benchmark, outperforming all\nrecent approaches.","url_abs":"http://arxiv.org/abs/1802.05451v4","url_pdf":"http://arxiv.org/pdf/1802.05451v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mapping-images-to-scene-graphs-with","repo_url":"https://github.com/shikorab/SceneGraph","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"scene-graph-classification","task_name":"Scene Graph Classification"},{"task_slug":"scene-graph-generation","task_name":"Scene Graph Generation"},{"task_slug":"structured-prediction","task_name":"Structured Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1802.05451","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}