{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-00523","title":"VIRTUE: Visual-Interactive Text-Image Universal Embedder","arxiv_id":"2510.00523","date":"2025-10-01","proceeding":"ICLR","authors":["Wei-Yao Wang","Kazuya Tateishi","Qiyu Wu","Shusuke Takahashi","Yuki Mitsufuji"],"abstract":"Multimodal representation learning models have demonstrated successful operation across complex tasks, and the integration of vision-language models (VLMs) has further enabled embedding models with instruction-following capabilities. However, existing embedding models lack visual-interactive capabilities to specify regions of interest from users (e.g., point, bounding box, mask), which have been explored in generative models to broaden their human-interactive applicability. Equipping embedding models with visual interactions not only would unlock new applications with localized grounding of user intent, which remains unexplored, but also enable the models to learn entity-level information within images to complement their global representations for conventional embedding tasks. In this paper, we propose a novel Visual-InteRactive Text-Image Universal Embedder (VIRTUE) that extends the capabilities of the segmentation model and the vision-language model to the realm of representation learning. In VIRTUE, the segmentation model can process visual prompts that pinpoint specific regions within an image, thereby enabling the embedder to handle complex and ambiguous scenarios more precisely. To evaluate the visual-interaction ability of VIRTUE, we introduce a large-scale Segmentation-and-Scene Caption Retrieval (SCaR) benchmark comprising 1M samples that aims to retrieve the text caption by jointly considering the entity with a specific object and image scene. VIRTUE consistently achieves a state-of-the-art performance with significant improvements across 36 universal MMEB (3.1%-8.5%) and five visual-interactive SCaR (15.2%-20.3%) tasks.","url_abs":"https://arxiv.org/abs/2510.00523","url_pdf":"https://arxiv.org/pdf/2510.00523","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.00523","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.00523"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/sony/virtue","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"found_in_text":{"samples":9,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5210e1b5ee25838a","entry":"batch_to_device","repo":"sony/virtue","repo_kind":"found_in_text","path":"codes/src/utils.py","file_url":"https://github.com/sony/virtue/blob/HEAD/codes/src/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5210e1b5ee25838a"}},{"code_sha256_prefix":"d3ca3a2a2eb093e7","entry":"cached","repo":"sony/virtue","repo_kind":"found_in_text","path":"codes/src/grad_cache/functional.py","file_url":"https://github.com/sony/virtue/blob/HEAD/codes/src/grad_cache/functional.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d3ca3a2a2eb093e7"}},{"code_sha256_prefix":"fab5db4632e2fa87","entry":"cat_input_tensor","repo":"sony/virtue","repo_kind":"found_in_text","path":"codes/src/grad_cache/functional.py","file_url":"https://github.com/sony/virtue/blob/HEAD/codes/src/grad_cache/functional.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fab5db4632e2fa87"}},{"code_sha256_prefix":"230ef02dd4dd96d9","entry":"dist_gather","repo":"sony/virtue","repo_kind":"found_in_text","path":"codes/src/dist_utils.py","file_url":"https://github.com/sony/virtue/blob/HEAD/codes/src/dist_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"230ef02dd4dd96d9"}},{"code_sha256_prefix":"89f1c28d66196e80","entry":"dist_gather_nograd","repo":"sony/virtue","repo_kind":"found_in_text","path":"codes/src/dist_utils.py","file_url":"https://github.com/sony/virtue/blob/HEAD/codes/src/dist_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"89f1c28d66196e80"}},{"code_sha256_prefix":"2ee2dcbdf5dce10a","entry":"find_latest_checkpoint","repo":"sony/virtue","repo_kind":"found_in_text","path":"codes/src/utils.py","file_url":"https://github.com/sony/virtue/blob/HEAD/codes/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2ee2dcbdf5dce10a"}},{"code_sha256_prefix":"7917a2079ec6b095","entry":"gather_input_tensor","repo":"sony/virtue","repo_kind":"found_in_text","path":"codes/src/grad_cache/functional.py","file_url":"https://github.com/sony/virtue/blob/HEAD/codes/src/grad_cache/functional.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7917a2079ec6b095"}},{"code_sha256_prefix":"edac44cad40a9c53","entry":"get_logger","repo":"sony/virtue","repo_kind":"found_in_text","path":"codes/src/logging.py","file_url":"https://github.com/sony/virtue/blob/HEAD/codes/src/logging.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"edac44cad40a9c53"}},{"code_sha256_prefix":"b1ad8cd3a7168b7f","entry":"varsize_gather_nograd","repo":"sony/virtue","repo_kind":"found_in_text","path":"codes/src/dist_utils.py","file_url":"https://github.com/sony/virtue/blob/HEAD/codes/src/dist_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1ad8cd3a7168b7f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.AI","source":"arxiv_api"},"syntology_extracted_results":null}