{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-19060","title":"PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions","arxiv_id":"2510.19060","date":"2025-10-21","proceeding":"ICLR","authors":["Amith Ananthram","Elias Stengel-Eskin","Lorena A. Bradford","Julia Demarest","Adam Purvis","Keith Krut","Robert Stein","Rina Elster Pantalony","Mohit Bansal","Kathleen McKeown"],"abstract":"While vision-language models (VLMs) have advanced into detailed image description, evaluation remains a challenge. Standard metrics (e.g. CIDEr, SPICE) were designed for short texts and tuned to recognize errors that are now uncommon, such as object misidentification. In contrast, long texts require sensitivity to attribute and relation attachments and scores that localize errors to particular text spans. In this work, we introduce PoSh, a metric for detailed image description that uses scene graphs as structured rubrics to guide LLMs-as-a-Judge, producing aggregate scores grounded in fine-grained errors (e.g. mistakes in compositional understanding). PoSh is replicable, interpretable and a better proxy for human raters than existing metrics (including GPT4o-as-a-Judge). To validate PoSh, we introduce a challenging new dataset, DOCENT. This novel benchmark contains artwork, paired with expert-written references, and model-generated descriptions, augmented with granular and coarse judgments of their quality from art history students. Thus, DOCENT enables evaluating both detailed image description metrics and detailed image description itself in a challenging new domain. We show that PoSh achieves stronger correlations (+0.05 Spearman $ρ$) with the human judgments in DOCENT than the best open-weight alternatives, is robust to image type (using CapArena, an existing dataset of web imagery) and is a capable reward function, outperforming standard supervised fine-tuning. Then, using PoSh, we characterize the performance of open and closed models in describing the paintings, sketches and statues in DOCENT and find that foundation models struggle to achieve full, error-free coverage of images with rich scene dynamics, establishing a demanding new task to gauge VLM progress. Through both PoSh and DOCENT, we hope to enable advances in important areas such as assistive text generation.","url_abs":"https://arxiv.org/abs/2510.19060","url_pdf":"https://arxiv.org/pdf/2510.19060","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2510.19060","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.19060"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/amith-ananthram/posh","reach":{"status":"ok"}}],"summary":{"unverified":6},"by_repo_kind":{"found_in_text":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"7fd5ec9660d43987","entry":"calculate_bootstrap_elo","repo":"amith-ananthram/posh","repo_kind":"found_in_text","path":"evaluate_posh_coarse.py","file_url":"https://github.com/amith-ananthram/posh/blob/HEAD/evaluate_posh_coarse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7fd5ec9660d43987"}},{"code_sha256_prefix":"b2b3a414133710eb","entry":"calculate_elo","repo":"amith-ananthram/posh","repo_kind":"found_in_text","path":"evaluate_posh_coarse.py","file_url":"https://github.com/amith-ananthram/posh/blob/HEAD/evaluate_posh_coarse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2b3a414133710eb"}},{"code_sha256_prefix":"8ae818ba09775810","entry":"create_visualization","repo":"amith-ananthram/posh","repo_kind":"found_in_text","path":"posh/visualizer/visualize.py","file_url":"https://github.com/amith-ananthram/posh/blob/HEAD/posh/visualizer/visualize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ae818ba09775810"}},{"code_sha256_prefix":"d1b63ea12f53af94","entry":"get_texts","repo":"amith-ananthram/posh","repo_kind":"found_in_text","path":"evaluate_posh_granular.py","file_url":"https://github.com/amith-ananthram/posh/blob/HEAD/evaluate_posh_granular.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1b63ea12f53af94"}},{"code_sha256_prefix":"b7ba305231182f37","entry":"load_coarse_benchmark","repo":"amith-ananthram/posh","repo_kind":"found_in_text","path":"evaluate_posh_coarse.py","file_url":"https://github.com/amith-ananthram/posh/blob/HEAD/evaluate_posh_coarse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7ba305231182f37"}},{"code_sha256_prefix":"8f3e9aebffda3949","entry":"save_visualization","repo":"amith-ananthram/posh","repo_kind":"found_in_text","path":"posh/visualizer/visualize.py","file_url":"https://github.com/amith-ananthram/posh/blob/HEAD/posh/visualizer/visualize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f3e9aebffda3949"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_api"},"syntology_extracted_results":null}