{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/incorporating-structured-representations-into","title":"Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs","arxiv_id":"2305.06343","date":"2023-05-10","proceeding":null,"authors":["Roei Herzig","Alon Mendelson","Leonid Karlinsky","Assaf Arbelle","Rogerio Feris","Trevor Darrell","Amir Globerson"],"abstract":"Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS) performance in a variety of tasks. However, recent works have shown that even the best VLMs struggle to capture aspects of compositional scene understanding, such as object attributes, relations, and action states. In contrast, obtaining structured annotations, such as scene graphs (SGs), that could improve these models is time-consuming and costly, and thus cannot be used on a large scale. Here we ask whether small SG datasets can provide sufficient information for enhancing structured understanding of pretrained VLMs. We show that it is indeed possible to improve VLMs when learning from SGs by integrating components that incorporate structured information into both visual and textual representations. For the visual side, we incorporate a special \"SG Component\" in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions that highlight different compositional aspects of the scene. Our method improves the performance of several popular VLMs on multiple VL datasets with only a mild degradation in ZS capabilities.","url_abs":"https://arxiv.org/abs/2305.06343v2","url_pdf":"https://arxiv.org/pdf/2305.06343v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"BLIP2 (SGVL)","rank_in_archive_order":25,"of":114,"metrics":{"Group Score":"23.3","Image Score":"28.5","Text Score":"42.8"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"BLIP (SGVL)","rank_in_archive_order":26,"of":114,"metrics":{"Group Score":"21.5","Image Score":"27.3","Text Score":"42.8"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"NegBLIP","rank_in_archive_order":29,"of":114,"metrics":{"Group Score":"18.5","Image Score":"24.0","Text Score":"42.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"BLIP2","rank_in_archive_order":32,"of":114,"metrics":{"Group Score":"19.0","Image Score":"23.8","Text Score":"42.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"NegBLIP2","rank_in_archive_order":34,"of":114,"metrics":{"Group Score":"20.5","Image Score":"26.0","Text Score":"41.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"BLIP (+Graph Text, +Graph Neg)","rank_in_archive_order":35,"of":114,"metrics":{"Group Score":"19.0","Image Score":"25.5","Text Score":"40.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"BLIP (+Graph Text)","rank_in_archive_order":36,"of":114,"metrics":{"Group Score":"16.5","Image Score":"20.5","Text Score":"40.3"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"BLIP","rank_in_archive_order":41,"of":114,"metrics":{"Group Score":"15.0","Image Score":"19.2","Text Score":"39.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"CLIP (SGVL)","rank_in_archive_order":61,"of":114,"metrics":{"Group Score":"9.8","Image Score":"14.0","Text Score":"32.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"NegCLIP","rank_in_archive_order":72,"of":114,"metrics":{"Group Score":"8.0","Image Score":"10.5","Text Score":"29.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"LLaVA","rank_in_archive_order":87,"of":114,"metrics":{"Group Score":"13.0","Image Score":"25.0","Text Score":"24.8"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"MiniGPT-4","rank_in_archive_order":91,"of":114,"metrics":{"Group Score":"9.5","Image Score":"18.0","Text Score":"23.3"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.06343","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}