{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-13399","title":"EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing","arxiv_id":"2509.13399","date":"2025-09-16","proceeding":null,"authors":["Tianyu Chen","Yasi Zhang","Zhi Zhang","Peiyu Yu","Shu Wang","Zhendong Wang","Kevin Lin","Xiaofei Wang","Zhengyuan Yang","Linjie Li","Chung-Ching Lin","Jianwen Xie","Oscar Leong","Lijuan Wang","Ying Nian Wu","Mingyuan Zhou"],"abstract":"Instruction-based image editing has advanced rapidly, yet reliable and interpretable evaluation remains a bottleneck. Current protocols either (i) depend on paired reference images, resulting in limited coverage and inheriting biases from prior generative models or (ii) rely solely on zero-shot vision language models (VLMs), whose prompt-based assessments of instruction following, content consistency, and visual quality are often imprecise. To address this, we introduce EdiVal, an automated and fine-grained evaluation framework grounded in an object-centric perspective, designed to assess not only standard single-turn but also multi-turn instruction-based editing with precision. Given an input image, EdiVal first decomposes it into semantically meaningful objects, then synthesizes diverse, context-aware editing instructions while dynamically updating object pools across turns. These two stages enable two novel object centric metrics tailored for multi turn evaluation and one global metric of visual quality: 1) EdiVal-IF, which measures instruction following by combining open vocabulary object detectors for symbolic checks with VLMs for semantic verification on detector guided crops; 2) EdiVal-CC, which evaluates content consistency by calculating semantic similarity of unchanged objects and background using the evolving object pools; and 3) EdiVal-VQ, which quantifies changes in overall visual quality with human preference models. Instantiating this pipeline, we build EdiVal Bench, a multi-turn editing benchmark covering 9 instruction types and 16 state-of-the-art editing models, spanning in-context, flow-matching, and diffusion paradigms. We demonstrate that EdiVal can be used to identify existing failure modes, thereby informing the development of the next generation of editing models.","url_abs":"https://arxiv.org/abs/2509.13399","url_pdf":"https://arxiv.org/pdf/2509.13399","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2509.13399","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.13399"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/TianyuCodings/EdiVal","reach":{"status":"ok"}}],"summary":{"unverified":6},"by_repo_kind":{"found_in_text":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"f05a8bb234c77122","entry":"main","repo":"TianyuCodings/EdiVal","repo_kind":"found_in_text","path":"baseline_generate/gpt4o.py","file_url":"https://github.com/TianyuCodings/EdiVal/blob/HEAD/baseline_generate/gpt4o.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f05a8bb234c77122"}},{"code_sha256_prefix":"efff3e70b7178982","entry":"process_one_image_multipass","repo":"TianyuCodings/EdiVal","repo_kind":"found_in_text","path":"baseline_generate/gpt4o.py","file_url":"https://github.com/TianyuCodings/EdiVal/blob/HEAD/baseline_generate/gpt4o.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efff3e70b7178982"}},{"code_sha256_prefix":"0b374c438b00d329","entry":"process_single_image_on_gpu","repo":"TianyuCodings/EdiVal","repo_kind":"found_in_text","path":"baseline_generate/flux.py","file_url":"https://github.com/TianyuCodings/EdiVal/blob/HEAD/baseline_generate/flux.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b374c438b00d329"}},{"code_sha256_prefix":"24582afb31f328ee","entry":"process_single_image_on_gpu","repo":"TianyuCodings/EdiVal","repo_kind":"found_in_text","path":"baseline_generate/ip2p.py","file_url":"https://github.com/TianyuCodings/EdiVal/blob/HEAD/baseline_generate/ip2p.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"24582afb31f328ee"}},{"code_sha256_prefix":"7de39aaaa6856872","entry":"process_single_image_on_gpu","repo":"TianyuCodings/EdiVal","repo_kind":"found_in_text","path":"baseline_generate/magic_brush.py","file_url":"https://github.com/TianyuCodings/EdiVal/blob/HEAD/baseline_generate/magic_brush.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7de39aaaa6856872"}},{"code_sha256_prefix":"101a46c578ccd912","entry":"run_multipass","repo":"TianyuCodings/EdiVal","repo_kind":"found_in_text","path":"baseline_generate/gpt4o.py","file_url":"https://github.com/TianyuCodings/EdiVal/blob/HEAD/baseline_generate/gpt4o.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"101a46c578ccd912"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_api"},"syntology_extracted_results":null}